Image processing apparatus and method for generating a restoration image

ABSTRACT

The present invention relates to anAn image processing apparatus which can restore, from a color and sensitivity mosaic image acquired using a CCD image sensor of the single plate type or the like, a color image signal of a wide dynamic range wherein the sensitivity characteristics of pixels are uniformized and each of the pixels has all of a plurality of color components is provided. A sensitivity uniformization section uniformizes the sensitivities of pixels of a color and sensitivity mosaic image to produce a color mosaic image, and a color interpolation section interpolates color components of the pixels of the color mosaic image M to produce output images R, G and B. The present inventionimage processing apparatus can be applied to a digital camera which converts a picked up optical image into a color image signal of a wide dynamic range.

CROSS REFERENCE TO RELATED APPLICATIONS

More than one reissue application has been filed for the reissue of U.S.Pat. No. 7,986,360. The present application is the parent application ofU.S. application Ser. No. 14/968,441, which is a reissue applicationthat was filed as a divisional application off of the presentapplication on Dec. 14, 2015. The present application is also a reissueof U.S. Pat. No. 7,986,360, filed on Apr. 30, 2008, which is acontinuation of U.S. patent application Ser. No. 10,466,015, filed onJan. 14, 2004, which was the National Stage of International ApplicationNo. PCT/JP02/00036, filed on Jan. 9, 2002, and which claims priority toJapanese Patent Document Nos. P2001-000979 filed on Jan. 9, 2001; andP2001-000980 filed on Jan. 9, 2001, the disclosures of which are hereinincorporated by reference.

BACKGROUND

This invention relates to an image processing apparatus, and moreparticularly to an image processing apparatus suitable for use forproduction of a color image signal of a wide dynamic range from an imagesignal acquired, for example, using a CCD image sensor of the singleplate type or the like.

A solid-state image pickup device such as a CCD (Charge Coupled Device)or a CMOS (Complementary Mental-Oxide Semiconductor) is utilized widelyin image pickup apparatus such as a video camera and a digital stillcamera, part inspection apparatus in the field of the FA (FactoryAutomation) and optical measuring instruments such as an electronicendoscope in the field of the ME (Medical Electronics).

Conventionally, a method is known wherein light intensity signalsmeasured with different sensitivities among different pixels aresynthesized in order to increase the dynamic range of image pickupapparatus and optical measuring instruments in which a solid-state imagepickup device is used. In the following, first to fourth related-artmethods of the type mentioned are described.

As the first related-art method, a method can be listed wherein incominglight beams branched to a plurality of optical axes having differentoptical transmission factors are measured by solid-state image pickupdevices disposed on the individual optical axes. This method isdisclosed in the official gazette of Japanese Patent Laid-Open No. Hei8-223491 and so forth. However, the first method has a problem in thatit is disadvantageous in terms of the reduction of the cost or thereduction of the space because it requires a plurality of solid-stateimage pickup devices and a complicated optical system for branchinglight.

As the second related-art method, a method can be listed wherein asingle solid-state image pickup device is used such that the exposuretime thereof is divided into a plurality of time periods to pick up aplurality of images and then the images are synthesized. This method isdisclosed in the official gazette of Japanese Patent Laid-Open No. Hei8-331461 and so forth. However, the second method has a problem in thatan image of a dynamic scene in which the intensity of light varies everymoment cannot be picked up properly because the information measuredwith the different sensitivities are picked up at different points oftime and with different time widths.

As the third related-art method, a method can be listed wherein a singlesolid-state image pickup device is used such that a plurality of lightreceiving elements adjacent each other on an image pickup face thereofform a set which corresponds to one pixel of an output image and havesensitivities different from each other to pick up an image. This methodis disclosed in the official gazette of U.S. Pat. No. 5,789,737. As amethod for making the sensitivities of light receiving elements whichform a solid-state image pickup device different from each other, amethod is available wherein the light receiving elements are coveredwith filters having transmission factors different from each other.Further, a technique which adapts the third related-art method to acolor image is disclosed in the official gazette of Japanese PatentLaid-Open No. 2000-69491.

The third related-art method is advantageous in terms of the reductionof the cost and the reduction of the space in terms of which the firstrelated-art method is disadvantageous. Further, the third related-artmethod can solve the problem of the second related-art method that animage of a dynamic scene cannot be picked up properly. However, with thethird related-art method, since a plurality of light receiving elementsadjacent each other form a set and correspond to one pixel of an outputimage, in order to secure a resolution of output pixels, a number ofimage pickup devices including a number of light receiving elementsequal to several times the number of pixels of the output image,resulting in a subject that a large unit cell size is required.

As the fourth related-art method, a method can be listed wherein animage pickup device having an ordinary dynamic range is used to pick upan image with a mechanism applied thereto which makes the exposuredifferent for each light receiving element corresponding to one pixel ofan output image and the resulting image signals are subject topredetermined image processing to produce an image signal of a widedynamic range. The mechanism for making the exposure different amongdifferent light receiving elements is implemented by producing a spatialsensitivity pattern by changing the light transmission factor or thenumerical aperture for each light receiving element. This method isdisclosed in a document ‘S. K. Nayar and T. Mitsunaga, “High DynamicRange Imaging: Spatially Varying Pixel Exposures”, Proc. of ComputerVision and Pattern Recognition 2000, Vol. 1, pp. 472-479, June, 2000’.

In the fourth related-art method, each of the light receiving elementshas only one kind of sensitivity. Consequently, each of pixels of animage picked up can acquire information of a dynamic range which theimage pickup device originally has. However, by applying predeterminedimage processing to resulting image signals so that the sensitivities ofall of the pixels may become equal to one another, an image having awide dynamic range can be produced. Further, since all of the lightreceiving elements are exposed to light at the same time, an image of asubject having some movement can be picked up properly. Furthermore,since one light receiving element corresponds to one pixel of outputimage, the problem that a great unit size is required does not occurwith the fourth related-art method.

As described above, the fourth related-art method can solve the problemsof the first to third related-art methods. However, the fourthrelated-art method has a premise that a monochromatic image is produced,and has a subject that a technique for producing a color image has notbeen established. More particularly, the fourth related-art method has asubject that a technique of producing image signals of all colorcomponents for all pixels from an image having different colors and/ordifferent sensitivities among different pixels and making thesensitivity uniform has not conventionally been established.

SUMMARY

The present invention has been made in such a situation as describedabove, and it is an object of the present invention to make it possibleto use a color and sensitivity mosaic image wherein the color and/or thesensitivity are different among different pixels to produce a restoredimage wherein the pixels have a uniformed sensitivity characteristic andeach pixel has all of a plurality of color components.

A first image processing apparatus of the present invention ischaracterized in that it includes restoration means for restoring, basedon a color and sensitivity mosaic image wherein each of a plurality ofpixels has one of a plurality of color components and one of a pluralityof sensitivity characteristics with respect to the intensity of lightand a plurality of ones of the pixels which have the same colorcomponent and the same sensitivity characteristic are arranged in agrating-like arrangement and besides a plurality of ones of the pixelswhich have the same color component irrespective of the sensitivitycharacteristic are arranged in a grating-like arrangement, a restorationimage wherein the sensitivities of the pixels are uniformized and eachof the pixels has all of the plurality of color components.

The restoration means may include luminance image production means forproducing a luminance image corresponding to the color and sensitivitymosaic image based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image and color mosaic patterninformation representative of an arrangement of the color components ofthe color and sensitivity mosaic image, and a plurality of monochromaticimage production means each for producing a monochromatic imagecorresponding to the color and sensitivity mosaic image based on thesensitivity mosaic pattern information, the color mosaic patterninformation and the luminance image.

The luminance image production means may include a plurality ofestimation means each for calculating an estimated value of a colorcomponent corresponding to each of the pixels of the color andsensitivity mosaic image, and luminance candidate value calculationmeans for calculating a luminance candidate value corresponding to eachof the pixels of the color and sensitivity mosaic image using aplurality of the estimated values calculated individually by theplurality of estimation means.

Each of the estimation means may calculate a plurality of estimatedvalue candidates individually corresponding to the plurality ofsensitivity characteristics, add the plurality of estimated valuecandidates and compensate for the non-linearity of the sensitivitycharacteristic appearing with the sum of the plurality of estimatedvalue candidates.

The luminance image production means may further include noise removalmeans for removing noise components of the luminance candidate value toproduce a luminance value.

Each of the monochromatic image production means may includemonochromatic image candidate production means for producing amonochromatic image candidate corresponding to the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information, and modification means for modifyingthe monochromatic image candidate based on the luminance image toproduce the monochromatic image.

The monochromatic image candidate production means may calculate aplurality of monochromatic candidate values individually correspondingto the plurality of sensitivity characteristics, add the plurality ofmonochromatic candidate values and compensate for the non-linearity ofthe sensitivity characteristic appearing with the sum of the pluralityof monochromatic candidate values to calculate pixel values of themonochromatic image candidate to produce the monochromatic imagecandidate.

The monochromatic image candidate production means may use a directionselective smoothing process to produce the monochromatic image candidatecorresponding to the color and sensitivity mosaic image.

The first image processing apparatus of the present invention mayfurther include image pickup means for picking up an image of a subjectto produce the color and sensitivity mosaic image.

The restoration means may include sensitivity characteristicuniformization means for uniformizing the sensitivity characteristics ofthe pixels based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image, and color interpolation meansfor interpolating color components of the pixels based on color mosaicpattern information representative of an arrangement of the colorcomponents of the color and sensitivity mosaic image.

The sensitivity characteristic uniformization means may uniformize thesensitivity characteristics of the pixels of the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information to produce a color mosaic image, andthe color interpolation means may interpolate the color components ofthe pixels of the color mosaic image based on the color mosaic patterninformation to produce the restoration image.

The sensitivity characteristic uniformization means may uniformize thesensitivity characteristics of the pixels of the color and sensitivitymosaic image without changing the kinds of the color components of thepixels of the color and sensitivity mosaic image based on thesensitivity mosaic pattern information and the color mosaic patterninformation to produce the color mosaic image.

The sensitivity characteristic uniformization means may uniformize thesensitivity characteristics of the pixels of the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information to produce the color mosaic image andupdate the color mosaic pattern information.

The sensitivity uniformization means may include compensation means forcompensating for the color components of the pixels of the color andsensitivity mosaic image based on the sensitivity mosaic patterninformation, discrimination means for discriminating the validity of thecolor components of the pixels of the color and sensitivity mosaic imagebased on the sensitivity mosaic pattern information, and modificationmeans for modifying the color components of the pixels compensated forby the compensation means through an interpolation process in responseto a result of the discrimination of the discrimination means.

The sensitivity uniformization means may include calculation means forcalculating estimated pixel values of the color and sensitivity mosaicimage based on the sensitivity mosaic pattern information, andcorrection means for correcting the estimated pixel values calculated bythe calculation means.

The color interpolation means may interpolate all of the colorcomponents of the pixels of the color and sensitivity mosaic imagewithout changing the sensitivity characteristics of the pixels based onthe sensitivity mosaic pattern information and the color mosaic patterninformation to produce a sensitivity mosaic image of the colorcomponents, and the sensitivity characteristic uniformization means mayuniformize the sensitivity characteristics of the pixels of thesensitivity mosaic image based on the sensitivity mosaic patterninformation to produce the restoration image.

The color interpolation means may include extraction means forextracting those of the pixels which have the same sensitivitycharacteristic from the color and sensitivity mosaic image, all colorcomponent interpolation means for interpolating all of the colorcomponents of the pixels extracted by the extraction means, andsynthesis means for synthesizing those of the pixels having all of thecolor components interpolated by the all color component interpolationmeans which have the same color component and have the differentsensitivity characteristics to produce the sensitivity mosaic image.

A first image processing method of the present invention ischaracterized in that it includes a restoration step of restoring, basedon a color and sensitivity mosaic image wherein each of a plurality ofpixels has one of a plurality of color components and one of a pluralityof sensitivity characteristics with respect to the intensity of lightand a plurality of ones of the pixels which have the same colorcomponent and the same sensitivity characteristic are arranged in agrating-like arrangement and besides a plurality of ones of the pixelswhich have the same color component irrespective of the sensitivitycharacteristic are arranged in a grating-like arrangement, a restorationimage wherein the sensitivities of the pixels are uniformized and eachof the pixels has all of the plurality of color components.

The restoration step may include a luminance image production step ofproducing a luminance image corresponding to the color and sensitivitymosaic image based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image and color mosaic patterninformation representative of an arrangement of the color components ofthe color and sensitivity mosaic image, and a plurality of monochromaticimage production steps each of producing a monochromatic imagecorresponding to the color and sensitivity mosaic image based on thesensitivity mosaic pattern information, the color mosaic patterninformation and the luminance image.

The luminance image production step may include a plurality ofestimation steps each of calculating an estimated value of a colorcomponent corresponding to each of the pixels of the color andsensitivity mosaic image, and a luminance candidate value calculationstep of calculating a luminance candidate value corresponding to each ofthe pixels of the color and sensitivity mosaic image using a pluralityof the estimated values calculated individually by the processing of theplurality of estimation steps.

The processing of each of the estimation steps may calculate a pluralityof estimated value candidate individually corresponding to the pluralityof sensitivity characteristics, add the plurality of estimated valuecandidates and compensate for the non-linearity of the sensitivitycharacteristic appearing with the sum of the plurality of estimatedvalue candidates.

The luminance image production step may further include a noise removalstep of removing noise components of the luminance candidate value toproduce a luminance value.

Each of the monochromatic image production steps may include amonochromatic image candidate production step of producing amonochromatic image candidate corresponding to the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information, and a modification step of modifyingthe monochromatic image candidate based on the luminance image toproduce the monochromatic image.

The processing of the monochromatic image candidate production step maycalculate a plurality of monochromatic candidate values individuallycorresponding to the plurality of sensitivity characteristics, add theplurality of monochromatic candidate values and compensate for thenon-linearity of the sensitivity characteristic appearing with the sumof the plurality of monochromatic candidate values to calculate pixelvalues of the monochromatic image candidate to produce the monochromaticimage candidate.

The processing of the monochromatic image candidate production step mayuse a direction selective smoothing process to produce the monochromaticimage candidate corresponding to the color and sensitivity mosaic image.

The first image processing method of the present invention may furtherinclude an image pickup step of picking up an image of a subject toproduce the color and sensitivity mosaic image.

The restoration step may include a sensitivity characteristicuniformization step of uniformizing the sensitivity characteristics ofthe pixels based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image, and a color interpolation stepof interpolating color components of the pixels based on color mosaicpattern information representative of an arrangement of the colorcomponents of the color and sensitivity mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce a colormosaic image, and the processing of the color interpolation step mayinterpolate the color components of the pixels of the color mosaic imagebased on the color mosaic pattern information to produce the restorationimage.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image without changing the kinds of the colorcomponents of the pixels of the color and sensitivity mosaic image basedon the sensitivity mosaic pattern information and the color mosaicpattern information to produce the color mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce thecolor mosaic image and update the color mosaic pattern information.

The sensitivity uniformization step may include a compensation step ofcompensating for the color components of the pixels of the color andsensitivity mosaic image based on the sensitivity mosaic patterninformation, a discrimination step of discriminating the validity of thecolor components of the pixels of the color and sensitivity mosaic imagebased on the sensitivity mosaic pattern information, and a modificationstep of modifying the color components of the pixels compensated for bythe processing of the compensation step through an interpolation processin response to a result of the discrimination of the discriminationmeans.

The sensitivity uniformization step may include a calculation step ofcalculating estimated pixel values of the color and sensitivity mosaicimage based on the sensitivity mosaic pattern information, and acorrection step of correcting the estimated pixel values calculated bythe processing of the calculation step.

The processing of the color interpolation step may interpolate all ofthe color components of the pixels of the color and sensitivity mosaicimage without changing the sensitivity characteristics of the pixelsbased on the sensitivity mosaic pattern information and the color mosaicpattern information to produce a sensitivity mosaic image of the colorcomponents, and the processing of the sensitivity characteristicuniformization step may uniformize the sensitivity characteristics ofthe pixels of the sensitivity mosaic image based on the sensitivitymosaic pattern information to produce the restoration image.

The color interpolation step may include an extraction step ofextracting those of the pixels which have the same sensitivitycharacteristic from the color and sensitivity mosaic image, an all colorcomponent interpolation step of interpolating all of the colorcomponents of the pixels extracted by the processing of the extractionstep, and a synthesis step of synthesizing those of the pixels havingall of the color components interpolated by the processing of the allcolor component interpolation step which have the same color componentand have the different sensitivity characteristics to produce thesensitivity mosaic image.

A program of a first recording medium of the present invention ischaracterized in that the program includes a restoration step ofrestoring, based on a color and sensitivity mosaic image wherein each ofa plurality of pixels has one of a plurality of color components and oneof a plurality of sensitivity characteristics with respect to theintensity of light and a plurality of ones of the pixels which have thesame color component and the same sensitivity characteristic arearranged in a grating-like arrangement and besides a plurality of onesof the pixels which have the same color component irrespective of thesensitivity characteristic are arranged in a grating-like arrangement, arestoration image wherein the sensitivities of the pixels areuniformized and each of the pixels has all of the plurality of colorcomponents.

The restoration step may include a luminance image production step ofproducing a luminance image corresponding to the color and sensitivitymosaic image based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image and color mosaic patterninformation representative of an arrangement of the color components ofthe color and sensitivity mosaic image, and a plurality of monochromaticimage production steps each of producing a monochromatic imagecorresponding to the color and sensitivity mosaic image based on thesensitivity mosaic pattern information, the color mosaic patterninformation and the luminance image.

The luminance image production step may include a plurality ofestimation steps each of calculating an estimated value of a colorcomponent corresponding to each of the pixels of the color andsensitivity mosaic image, and a luminance candidate value calculationstep of calculating a luminance candidate value corresponding to each ofthe pixels of the color and sensitivity mosaic image using a pluralityof the estimated values calculated individually by the processing of theplurality of estimation steps.

The processing of each of the estimation steps may calculate a pluralityof estimated value candidates individually corresponding to theplurality of sensitivity characteristics, add the plurality of estimatedvalue candidates and compensate for the non-linearity of the sensitivitycharacteristic appearing with the sum of the plurality of estimatedvalue candidates.

The luminance image production step may further include a noise removalstep of removing noise components of the luminance candidate value toproduce a luminance value.

Each of the monochromatic image production steps may include amonochromatic image candidate production step of producing amonochromatic image candidate corresponding to the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information, and a modification step of modifyingthe monochromatic image candidate based on the luminance image toproduce the monochromatic image.

The processing of the monochromatic image candidate production step maycalculate a plurality of monochromatic candidate values individuallycorresponding to the plurality of sensitivity characteristics, add theplurality of monochromatic candidate values and compensate for thenon-linearity of the sensitivity characteristic appearing with the sumof the plurality of monochromatic candidate values to calculate pixelvalues of the monochromatic image candidate to produce the monochromaticimage candidate.

The processing of the monochromatic image candidate production step mayuse a direction selective smoothing process to produce the monochromaticimage candidate corresponding to the color and sensitivity mosaic image.

The program of the first recording medium of the present invention mayfurther include an image pickup controlling step of controlling aprocess of picking up an image of a subject to produce the color andsensitivity mosaic image.

The restoration step may include a sensitivity characteristicuniformization step of uniformizing the sensitivity characteristics ofthe pixels based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image, and a color interpolation stepof interpolating color components of the pixels based on color mosaicpattern information representative of an arrangement of the colorcomponents of the color and sensitivity mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce a colormosaic image, and the processing of the color interpolation step mayinterpolate the color components of the pixels of the color mosaic imagebased on the color mosaic pattern information to produce the restorationimage.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image without changing the kinds of the colorcomponents of the pixels of the color and sensitivity mosaic image basedon the sensitivity mosaic pattern information and the color mosaicpattern information to produce the color mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce thecolor mosaic image and update the color mosaic pattern information.

The sensitivity uniformization step may include a compensation step ofcompensating for the color components of the pixels of the color andsensitivity mosaic image based on the sensitivity mosaic patterninformation, a discrimination step of discriminating the validity of thecolor components of the pixels of the color and sensitivity mosaic imagebased on the sensitivity mosaic pattern information, and a modificationstep of modifying the color components of the pixels compensated for bythe processing of the compensation step through an interpolation processin response to a result of the discrimination of the discriminationmeans.

The sensitivity uniformization step may include a calculation step ofcalculating estimated pixel values of the color and sensitivity mosaicimage based on the sensitivity mosaic pattern information, and acorrection step of correcting the estimated pixel values calculated bythe processing of the calculation step.

The processing of the color interpolation step may interpolate all ofthe color components of the pixels of the color and sensitivity mosaicimage without changing the sensitivity characteristics of the pixelsbased on the sensitivity mosaic pattern information and the color mosaicpattern information to produce a sensitivity mosaic image of the colorcomponents, and the processing of the sensitivity characteristicuniformization step may uniformize the sensitivity characteristics ofthe pixels of the sensitivity mosaic image based on the sensitivitymosaic pattern information to produce the restoration image.

The color interpolation step may include an extraction step ofextracting those of the pixels which have the same sensitivitycharacteristic from the color and sensitivity mosaic image, an all colorcomponent interpolation step of interpolating all of the colorcomponents of the pixels extracted by the processing of the extractionstep, and a synthesis step of synthesizing those of the pixels havingall of the color components interpolated by the processing of the allcolor component interpolation step which have the same color componentand have the different sensitivity characteristics to produce thesensitivity mosaic image.

A first program of the present invention is characterized in that itcauses a computer to execute a restoration step of restoring, based on acolor and sensitivity mosaic image wherein each of a plurality of pixelshas one of a plurality of color components and one of a plurality ofsensitivity characteristics with respect to the intensity of light and aplurality of ones of the pixels which have the same color component andthe same sensitivity characteristic are arranged in a grating-likearrangement and besides a plurality of ones of the pixels which have thesame color component irrespective of the sensitivity characteristic arearranged in a grating-like arrangement, a restoration image wherein thesensitivities of the pixels are uniformized and each of the pixels hasall of the plurality of color components.

The restoration step may include a luminance image production step ofproducing a luminance image corresponding to the color and sensitivitymosaic image based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image and color mosaic patterninformation representative of an arrangement of the color components ofthe color and sensitivity mosaic image, and a plurality of monochromaticimage production steps each of producing a monochromatic imagecorresponding to the color and sensitivity mosaic image based on thesensitivity mosaic pattern information, the color mosaic patterninformation and the luminance image.

The luminance image production step may include a plurality ofestimation steps each of calculating an estimated value of a colorcomponent corresponding to each of the pixels of the color andsensitivity mosaic image, and a luminance candidate value calculationstep of calculating a luminance candidate value corresponding to each ofthe pixels of the color and sensitivity mosaic image using a pluralityof the estimated values calculated individually by the processing of theplurality of estimation steps.

The processing of each of the estimation steps may calculate a pluralityof estimated value candidates individually corresponding to theplurality of sensitivity characteristics, add the plurality of estimatedvalue candidates and compensate for the non-linearity of the sensitivitycharacteristic appearing with the sum of the plurality of estimatedvalue candidates.

The luminance image production step may further include a noise removalstep of removing noise components of the luminance candidate value toproduce a luminance value.

Each of the monochromatic image production steps may include amonochromatic image candidate production step of producing amonochromatic image candidate corresponding to the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information, and a modification step of modifyingthe monochromatic image candidate based on the luminance image toproduce the monochromatic image.

The processing of the monochromatic image candidate production step maycalculate a plurality of monochromatic candidate values individuallycorresponding to the plurality of sensitivity characteristics, add theplurality of monochromatic candidate values and compensate for thenon-linearity of the sensitivity characteristic appearing with the sumof the plurality of monochromatic candidate values to calculate pixelvalues of the monochromatic image candidate to produce the monochromaticimage candidate.

The processing of the monochromatic image candidate production step mayuse a direction selective smoothing process to produce the monochromaticimage candidate corresponding to the color and sensitivity mosaic image.

The first program of the present invention may further include an imagepickup controlling step of controlling a process of picking up an imageof a subject to produce the color and sensitivity mosaic image.

The restoration step may include a sensitivity characteristicuniformization step of uniformizing the sensitivity characteristics ofthe pixels based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image, and a color interpolation stepof interpolating color components of the pixels based on color mosaicpattern information representative of an arrangement of the colorcomponents of the color and sensitivity mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce a colormosaic image, and the processing of the color interpolation step mayinterpolate the color components of the pixels of the color mosaic imagebased on the color mosaic pattern information to produce the restorationimage.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image without changing the kinds of the colorcomponents of the pixels of the color and sensitivity mosaic image basedon the sensitivity mosaic pattern information and the color mosaicpattern information to produce the color mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce thecolor mosaic image and update the color mosaic pattern information.

The sensitivity uniformization step may include a compensation step ofcompensating for the color components of the pixels of the color andsensitivity mosaic image based on the sensitivity mosaic patterninformation, a discrimination step of discriminating the validity of thecolor components of the pixels of the color and sensitivity mosaic imagebased on the sensitivity mosaic pattern information, and a modificationstep of modifying the color components of the pixels compensated for bythe processing of the compensation step through an interpolation processin response to a result of the discrimination of the discriminationmeans.

The sensitivity uniformization step may include a calculation step ofcalculating estimated pixel values of the color and sensitivity mosaicimage based on the sensitivity mosaic pattern information, and acorrection step of correcting the estimated pixel values calculated bythe processing of the calculation step.

The processing of the color interpolation step may interpolate all ofthe color components of the pixels of the color and sensitivity mosaicimage without changing the sensitivity characteristics of the pixelsbased on the sensitivity mosaic pattern information and the color mosaicpattern information to produce a sensitivity mosaic image of the colorcomponents, and the processing of the sensitivity characteristicuniformization step may uniformize the sensitivity characteristics ofthe pixels of the sensitivity mosaic image based on the sensitivitymosaic pattern information to produce the restoration image.

The color interpolation step may include an extraction step ofextracting those of the pixels which have the same sensitivitycharacteristic from the color and sensitivity mosaic image, an all colorcomponent interpolation step of interpolating all of the colorcomponents of the pixels extracted by the processing of the extractionstep, and a synthesis step of synthesizing those of the pixels havingall of the color components interpolated by the processing of the allcolor component interpolation step which have the same color componentand have the different sensitivity characteristics to produce thesensitivity mosaic image.

A second image processing apparatus of the present invention ischaracterized in that it includes restoration means for restoring, basedon a color and sensitivity mosaic image wherein each of a plurality ofpixels has one of a plurality of color components and one of a pluralityof sensitivity characteristics with respect to the intensity of lightand a plurality of ones of the pixels which have the same colorcomponent and the same sensitivity characteristic are arranged in agrating-like arrangement and besides a plurality of ones of the pixelswhich have the same sensitivity characteristic irrespective of the colorcomponent are arranged in a grating-like arrangement such that totaling5 pixels including an arbitrary pixel and four pixels neighboringupwardly, downwardly, leftwardly and rightwardly of the arbitrary pixelinclude all of the color components, a restoration image wherein thesensitivities of the pixels are uniformized and each of the pixels hasall of the plurality of color components.

The restoration means may includes luminance image production means forproducing a luminance image corresponding to the color and sensitivitymosaic image based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image and color mosaic patterninformation representative of an arrangement of the color components ofthe color and sensitivity mosaic image, and a plurality of monochromaticimage production means each for producing a monochromatic imagecorresponding to the color and sensitivity mosaic image based on thesensitivity mosaic pattern information, the color mosaic patterninformation and the luminance image.

The luminance image production means may include a plurality ofestimation means each for calculating an estimated value of a colorcomponent corresponding to each of the pixels of the color andsensitivity mosaic image, and luminance candidate value calculationmeans for calculating a luminance candidate value corresponding to eachof the pixels of the color and sensitivity mosaic image using aplurality of the estimated values calculated individually by theplurality of estimation means.

Each of the estimation means may calculate a plurality of estimatedvalue candidates individually corresponding to the plurality ofsensitivity characteristics, add the plurality of estimated valuecandidates and compensate for the non-linearity of the sensitivitycharacteristic appearing with the sum of the plurality of estimatedvalue candidates.

The luminance image production means may further include noise removalmeans for removing noise components of the luminance candidate value toproduce a luminance value.

Each of the monochromatic image production means may includemonochromatic image candidate production means for producing amonochromatic image candidate corresponding to the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information, and modification means for modifyingthe monochromatic image candidate based on the luminance image toproduce the monochromatic image.

The monochromatic image candidate production means may calculate aplurality of monochromatic candidate values individually correspondingto the plurality of sensitivity characteristics, add the plurality ofmonochromatic candidate values and compensate for the non-linearity ofthe sensitivity characteristic appearing with the sum of the pluralityof monochromatic candidate values to calculate pixel values of themonochromatic image candidate to produce the monochromatic imagecandidate.

The monochromatic image candidate production means may use a directionselective smoothing process to produce the monochromatic image candidatecorresponding to the color and sensitivity mosaic image.

The second image processing apparatus of the present invention mayfurther include image pickup means for picking up an image of a subjectto produce the color and sensitivity mosaic image.

The restoration means may include sensitivity characteristicuniformization means for uniformizing the sensitivity characteristics ofthe pixels based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image, and color interpolation meansfor interpolating color components of the pixels based on color mosaicpattern information representative of an arrangement of the colorcomponents of the color and sensitivity mosaic image.

The sensitivity characteristic uniformization means may uniformize thesensitivity characteristics of the pixels of the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information to produce a color mosaic image, andthe color interpolation means may interpolate the color components ofthe pixels of the color mosaic image based on the color mosaic patterninformation to produce the restoration image.

The sensitivity characteristic uniformization means may uniformize thesensitivity characteristics of the pixels of the color and sensitivitymosaic image without changing the kinds of the color components of thepixels of the color and sensitivity mosaic image based on thesensitivity mosaic pattern information and the color mosaic patterninformation to produce the color mosaic image.

The sensitivity characteristic uniformization means may uniformize thesensitivity characteristics of the pixels of the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information to produce the color mosaic image andupdate the color mosaic pattern information.

The sensitivity uniformization means may include compensation means forcompensating for the color components of the pixels of the color andsensitivity mosaic image based on the sensitivity mosaic patterninformation, discrimination means for discriminating the validity of thecolor components of the pixels of the color and sensitivity mosaic imagebased on the sensitivity mosaic pattern information, and modificationmeans for modifying the color components of the pixels compensated forby the compensation means through an interpolation process in responseto a result of the discrimination of the discrimination means.

The sensitivity uniformization means may include calculation means forcalculating estimated pixel values of the color and sensitivity mosaicimage based on the sensitivity mosaic pattern information, andcorrection means for correcting the estimated pixel values calculated bythe calculation means.

The color interpolation means may interpolate all of the colorcomponents of the pixels of the color and sensitivity mosaic imagewithout changing the sensitivity characteristics of the pixels based onthe sensitivity mosaic pattern information and the color mosaic patterninformation to produce a sensitivity mosaic image of the colorcomponents, and the sensitivity characteristic uniformization means mayuniformize the sensitivity characteristics of the pixels of thesensitivity mosaic image based on the sensitivity mosaic patterninformation to produce the restoration image.

The color interpolation means may include extraction means forextracting those of the pixels which have the same sensitivitycharacteristic from the color and sensitivity mosaic image, all colorcomponent interpolation means for interpolating all of the colorcomponents of the pixels extracted by the extraction means, andsynthesis means for synthesizing those of the pixels having all of thecolor components interpolated by the all color component interpolationmeans which have the same color component and have the differentsensitivity characteristics to produce the sensitivity mosaic image.

A second image processing method of the present invention ischaracterized in that it includes a restoration step of restoring, basedon a color and sensitivity mosaic image wherein each of a plurality ofpixels has one of a plurality of color components and one of a pluralityof sensitivity characteristics with respect to the intensity of lightand a plurality of ones of the pixels which have the same colorcomponent and the same sensitivity characteristic are arranged in agrating-like arrangement and besides a plurality of ones of the pixelswhich have the same sensitivity characteristic irrespective of the colorcomponent are arranged in a grating-like arrangement such that totaling5 pixels including an arbitrary pixel and four pixels neighboringupwardly, downwardly, leftwardly and rightwardly of the arbitrary pixelinclude all of the color components, a restoration image wherein thesensitivities of the pixels are uniformized and each of the pixels hasall of the plurality of color components.

The restoration step may include a luminance image production step ofproducing a luminance image corresponding to the color and sensitivitymosaic image based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image and color mosaic patterninformation representative of an arrangement of the color components ofthe color and sensitivity mosaic image, and a plurality of monochromaticimage production steps each of producing a monochromatic imagecorresponding to the color and sensitivity mosaic image based on thesensitivity mosaic pattern information, the color mosaic patterninformation and the luminance image.

The luminance image production step may include a plurality ofestimation steps each of calculating an estimated value of a colorcomponent corresponding to each of the pixels of the color andsensitivity mosaic image, and a luminance candidate value calculationstep of calculating a luminance candidate value corresponding to each ofthe pixels of the color and sensitivity mosaic image using a pluralityof the estimated values calculated individually by the processing of theplurality of estimation steps.

The processing of each of the estimation steps may calculate a pluralityof estimated value candidates individually corresponding to theplurality of sensitivity characteristics, add the plurality of estimatedvalue candidates and compensate for the non-linearity of the sensitivitycharacteristic appearing with the sum of the plurality of estimatedvalue candidates.

The luminance image production step may further include a noise removalstep of removing noise components of the luminance candidate value toproduce a luminance value.

Each of the monochromatic image production steps may include amonochromatic image candidate production step of producing amonochromatic image candidate corresponding to the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information, and a modification step of modifyingthe monochromatic image candidate based on the luminance image toproduce the monochromatic image.

The processing of the monochromatic image candidate production step maycalculate a plurality of monochromatic candidate values individuallycorresponding to the plurality of sensitivity characteristics, add theplurality of monochromatic candidate values and compensate for thenon-linearity of the sensitivity characteristic appearing with the sumof the plurality of monochromatic candidate values to calculate pixelvalues of the monochromatic image candidate to produce the monochromaticimage candidate.

The processing of the monochromatic image candidate production step mayuse a direction selective smoothing process to produce the monochromaticimage candidate corresponding to the color and sensitivity mosaic image.

The second image processing method of the present invention may furtherinclude an image pickup step of picking up an image of a subject toproduce the color and sensitivity mosaic image.

The restoration step may include a sensitivity characteristicuniformization step of uniformizing the sensitivity characteristics ofthe pixels based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image, and a color interpolation stepof interpolating color components of the pixels based on color mosaicpattern information representative of an arrangement of the colorcomponents of the color and sensitivity mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce a colormosaic image, and the processing of the color interpolation step mayinterpolate the color components of the pixels of the color mosaic imagebased on the color mosaic pattern information to produce the restorationimage.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image without changing the kinds of the colorcomponents of the pixels of the color and sensitivity mosaic image basedon the sensitivity mosaic pattern information and the color mosaicpattern information to produce the color mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce thecolor mosaic image and update the color mosaic pattern information.

The sensitivity uniformization step may include a compensation step ofcompensating for the color components of the pixels of the color andsensitivity mosaic image based on the sensitivity mosaic patterninformation, a discrimination step of discriminating the validity of thecolor components of the pixels of the color and sensitivity mosaic imagebased on the sensitivity mosaic pattern information, and a modificationstep of modifying the color components of the pixels compensated for bythe processing of the compensation step through an interpolation processin response to a result of the discrimination of the discriminationmeans.

The sensitivity uniformization step may include a calculation step ofcalculating estimated pixel values of the color and sensitivity mosaicimage based on the sensitivity mosaic pattern information, and acorrection step of correcting the estimated pixel values calculated bythe processing of the calculation step.

The processing of the color interpolation step may interpolate all ofthe color components of the pixels of the color and sensitivity mosaicimage without changing the sensitivity characteristics of the pixelsbased on the sensitivity mosaic pattern information and the color mosaicpattern information to produce a sensitivity mosaic image of the colorcomponents, and the processing of the sensitivity characteristicuniformization step may uniformize the sensitivity characteristics ofthe pixels of the sensitivity mosaic image based on the sensitivitymosaic pattern information to produce the restoration image.

The color interpolation step may include an extraction step ofextracting those of the pixels which have the same sensitivitycharacteristic from the color and sensitivity mosaic image, an all colorcomponent interpolation step of interpolating all of the colorcomponents of the pixels extracted by the processing of the extractionstep, and a synthesis step of synthesizing those of the pixels havingall of the color components interpolated by the processing of the allcolor component interpolation step which have the same color componentand have the different sensitivity characteristics to produce thesensitivity mosaic image.

A program of a second recording medium of the present invention ischaracterized in that the program includes a restoration step ofrestoring, based on a color and sensitivity mosaic image wherein each ofa plurality of pixels has one of a plurality of color components and oneof a plurality of sensitivity characteristics with respect to theintensity of light and a plurality of ones of the pixels which have thesame color component and the same sensitivity characteristic arearranged in a grating-like arrangement and besides a plurality of onesof the pixels which have the same sensitivity characteristicirrespective of the color component are arranged in a grating-likearrangement such that totaling 5 pixels including an arbitrary pixel andfour pixels neighboring upwardly, downwardly, leftwardly and rightwardlyof the arbitrary pixel include all of the color components, arestoration image wherein the sensitivities of the pixels areuniformized and each of the pixels has all of the plurality of colorcomponents.

The restoration step may include a luminance image production step ofproducing a luminance image corresponding to the color and sensitivitymosaic image based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image and color mosaic patterninformation representative of an arrangement of the color components ofthe color and sensitivity mosaic image, and a plurality of monochromaticimage production steps each of producing a monochromatic imagecorresponding to the color and sensitivity mosaic image based on thesensitivity mosaic pattern information, the color mosaic patterninformation and the luminance image.

The luminance image production step may include a plurality ofestimation steps each of calculating an estimated value of a colorcomponent corresponding to each of the pixels of the color andsensitivity mosaic image, and a luminance candidate value calculationstep of calculating a luminance candidate value corresponding to each ofthe pixels of the color and sensitivity mosaic image using a pluralityof the estimated values calculated individually by the processing of theplurality of estimation steps.

The processing of each of the estimation steps may calculate a pluralityof estimated value candidates individually corresponding to theplurality of sensitivity characteristics, add the plurality of estimatedvalue candidates and compensate for the non-linearity of the sensitivitycharacteristic appearing with the sum of the plurality of estimatedvalue candidates.

The luminance image production step may further include a noise removalstep of removing noise components of the luminance candidate value toproduce a luminance value.

Each of the monochromatic image production steps may include amonochromatic image candidate production step of producing amonochromatic image candidate corresponding to the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information, and a modification step of modifyingthe monochromatic image candidate based on the luminance image toproduce the monochromatic image.

The processing of the monochromatic image candidate production step maycalculate a plurality of monochromatic candidate values individuallycorresponding to the plurality of sensitivity characteristics, add theplurality of monochromatic candidate values and compensate for thenon-linearity of the sensitivity characteristic appearing with the sumof the plurality of monochromatic candidate values to calculate pixelvalues of the monochromatic image candidate to produce the monochromaticimage candidate.

The processing of the monochromatic image candidate production step mayuse a direction selective smoothing process to produce the monochromaticimage candidate corresponding to the color and sensitivity mosaic image.

The program of the second recording medium of the present invention mayfurther include an image pickup controlling step of controlling aprocess of picking up an image of a subject to produce the color andsensitivity mosaic image.

The restoration step may include a sensitivity characteristicuniformization step of uniformizing the sensitivity characteristics ofthe pixels based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image, and a color interpolation stepof interpolating color components of the pixels based on color mosaicpattern information representative of an arrangement of the colorcomponents of the color and sensitivity mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce a colormosaic image, and the processing of the color interpolation step mayinterpolate the color components of the pixels of the color mosaic imagebased on the color mosaic pattern information to produce the restorationimage.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image without changing the kinds of the colorcomponents of the pixels of the color and sensitivity mosaic image basedon the sensitivity mosaic pattern information and the color mosaicpattern information to produce the color mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce thecolor mosaic image and update the color mosaic pattern information.

The sensitivity uniformization step may include a compensation step ofcompensating for the color components of the pixels of the color andsensitivity mosaic image based on the sensitivity mosaic patterninformation, a discrimination step of discriminating the validity of thecolor components of the pixels of the color and sensitivity mosaic imagebased on the sensitivity mosaic pattern information, and a modificationstep of modifying the color components of the pixels compensated for bythe processing of the compensation step through an interpolation processin response to a result of the discrimination of the discriminationmeans.

The sensitivity uniformization step may include a calculation step ofcalculating estimated pixel values of the color and sensitivity mosaicimage based on the sensitivity mosaic pattern information, and acorrection step of correcting the estimated pixel values calculated bythe processing of the calculation step.

The processing of the color interpolation step may interpolate all ofthe color components of the pixels of the color and sensitivity mosaicimage without changing the sensitivity characteristics of the pixelsbased on the sensitivity mosaic pattern information and the color mosaicpattern information to produce a sensitivity mosaic image of the colorcomponents, and the processing of the sensitivity characteristicuniformization step may uniformize the sensitivity characteristics ofthe pixels of the sensitivity mosaic image based on the sensitivitymosaic pattern information to produce the restoration image.

The color interpolation step may include an extraction step ofextracting those of the pixels which have the same sensitivitycharacteristic from the color and sensitivity mosaic image, an all colorcomponent interpolation step of interpolating all of the colorcomponents of the pixels extracted by the processing of the extractionstep, and a synthesis step of synthesizing those of the pixels havingall of the color components interpolated by the processing of the allcolor component interpolation step which have the same color componentand have the different sensitivity characteristics to produce thesensitivity mosaic image.

A second program of the present invention is characterized in that itcauses a computer to execute a restoration step of restoring, based on acolor and sensitivity mosaic image wherein each of a plurality of pixelshas one of a plurality of color components and one of a plurality ofsensitivity characteristics with respect to the intensity of light and aplurality of ones of the pixels which have the same color component andthe same sensitivity characteristic are arranged in a grating-likearrangement and besides a plurality of ones of the pixels which have thesame sensitivity characteristic irrespective of the color component arearranged in a grating-like arrangement such that totaling 5 pixelsincluding an arbitrary pixel and four pixels neighboring upwardly,downwardly, leftwardly and rightwardly of the arbitrary pixel includeall of the color components, a restoration image wherein thesensitivities of the pixels are uniformized and each of the pixels hasall of the plurality of color components.

The restoration step may include a luminance image production step ofproducing a luminance image corresponding to the color and sensitivitymosaic image based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image and color mosaic patterninformation representative of an arrangement of the color components ofthe color and sensitivity mosaic image, and a plurality of monochromaticimage production steps each of producing a monochromatic imagecorresponding to the color and sensitivity mosaic image based on thesensitivity mosaic pattern information, the color mosaic patterninformation and the luminance image.

The luminance image production step may include a plurality ofestimation steps each of calculating an estimated value of a colorcomponent corresponding to each of the pixels of the color andsensitivity mosaic image, and a luminance candidate value calculationstep of calculating a luminance candidate value corresponding to each ofthe pixels of the color and sensitivity mosaic image using a pluralityof the estimated values calculated individually by the processing of theplurality of estimation steps.

The processing of each of the estimation steps may calculate a pluralityof estimated value candidates individually corresponding to theplurality of sensitivity characteristics, add the plurality of estimatedvalue candidates and compensate for the non-linearity of the sensitivitycharacteristic appearing with the sum of the plurality of estimatedvalue candidates.

The luminance image production step may further include a noise removalstep of removing noise components of the luminance candidate value toproduce a luminance value.

Each of the monochromatic image production steps may include amonochromatic image candidate production step of producing amonochromatic image candidate corresponding to the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information, and a modification step of modifyingthe monochromatic image candidate based on the luminance image toproduce the monochromatic image.

The processing of the monochromatic image candidate production step maycalculate a plurality of monochromatic candidate values individuallycorresponding to the plurality of sensitivity characteristics, add theplurality of monochromatic candidate values and compensate for thenon-linearity of the sensitivity characteristic appearing with the sumof the plurality of monochromatic candidate values to calculate pixelvalues of the monochromatic image candidate to produce the monochromaticimage candidate.

The processing of the monochromatic image candidate production step mayuse a direction selective smoothing process to produce the monochromaticimage candidate corresponding to the color and sensitivity mosaic image.

The second program of the present invention may further include an imagepickup controlling step of controlling a process of picking up an imageof a subject to produce the color and sensitivity mosaic image.

The restoration step may include a sensitivity characteristicuniformization step of uniformizing the sensitivity characteristics ofthe pixels based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image, and a color interpolation stepof interpolating color components of the pixels based on color mosaicpattern information representative of an arrangement of the colorcomponents of the color and sensitivity mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce a colormosaic image, and the processing of the color interpolation step mayinterpolate the color components of the pixels of the color mosaic imagebased on the color mosaic pattern information to produce the restorationimage.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image without changing the kinds of the colorcomponents of the pixels of the color and sensitivity mosaic image basedon the sensitivity mosaic pattern information and the color mosaicpattern information to produce the color mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce thecolor mosaic image and update the color mosaic pattern information.

The sensitivity uniformization step may include a compensation step ofcompensating for the color components of the pixels of the color andsensitivity mosaic image based on the sensitivity mosaic patterninformation, a discrimination step of discriminating the validity of thecolor components of the pixels of the color and sensitivity mosaic imagebased on the sensitivity mosaic pattern information, and a modificationstep of modifying the color components of the pixels compensated for bythe processing of the compensation step through an interpolation processin response to a result of the discrimination of the discriminationmeans.

The sensitivity uniformization step may include a calculation step ofcalculating estimated pixel values of the color and sensitivity mosaicimage based on the sensitivity mosaic pattern information, and acorrection step of correcting the estimated pixel values calculated bythe processing of the calculation step.

The processing of the color interpolation step may interpolate all ofthe color components of the pixels of the color and sensitivity mosaicimage without changing the sensitivity characteristics of the pixelsbased on the sensitivity mosaic pattern information and the color mosaicpattern information to produce a sensitivity mosaic image of the colorcomponents, and the processing of the sensitivity characteristicuniformization step may uniformize the sensitivity characteristics ofthe pixels of the sensitivity mosaic image based on the sensitivitymosaic pattern information to produce the restoration image.

The color interpolation step may include an extraction step ofextracting those of the pixels which have the same sensitivitycharacteristic from the color and sensitivity mosaic image, an all colorcomponent interpolation step of interpolating all of the colorcomponents of the pixels extracted by the processing of the extractionstep, and a synthesis step of synthesizing those of the pixels havingall of the color components interpolated by the processing of the allcolor component interpolation step which have the same color componentand have the different sensitivity characteristics to produce thesensitivity mosaic image.

A third image processing apparatus of the present invention ischaracterized in that it includes sensitivity characteristicuniformization means for uniformizing the sensitivity characteristics ofthe pixels based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image, and color interpolation meansfor interpolating color components of the pixels based on color mosaicpattern information representative of an arrangement of the colorcomponents of the color and sensitivity mosaic image.

The sensitivity characteristic uniformization means may uniformize thesensitivity characteristics of the pixels of the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information to produce a color mosaic image, andthe color interpolation means may interpolate the color components ofthe pixels of the color mosaic image based on the color mosaic patterninformation to produce the restoration image.

The sensitivity characteristic uniformization means may uniformize thesensitivity characteristics of the pixels of the color and sensitivitymosaic image without changing the kinds of the color components of thepixels of the color and sensitivity mosaic image based on thesensitivity mosaic pattern information and the color mosaic patterninformation to produce the color mosaic image.

The sensitivity characteristic uniformization means may uniformize thesensitivity characteristics of the pixels of the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information to produce the color mosaic image andupdate the color mosaic pattern information.

The sensitivity uniformization means may include compensation means forcompensating for the color components of the pixels of the color andsensitivity mosaic image based on the sensitivity mosaic patterninformation, discrimination means for discriminating the validity of thecolor components of the pixels of the color and sensitivity mosaic imagebased on the sensitivity mosaic pattern information, and modificationmeans for modifying the color components of the pixels compensated forby the compensation means through an interpolation process in responseto a result of the discrimination of the discrimination means.

The sensitivity uniformization means may include calculation means forcalculating estimated pixel values of the color and sensitivity mosaicimage based on the sensitivity mosaic pattern information, andcorrection means for correcting the estimated pixel values calculated bythe calculation means.

The color interpolation means may interpolate all of the colorcomponents of the pixels of the color and sensitivity mosaic imagewithout changing the sensitivity characteristics of the pixels based onthe sensitivity mosaic pattern information and the color mosaic patterninformation to produce a sensitivity mosaic image of the colorcomponents, and the sensitivity characteristic uniformization means mayuniformize the sensitivity characteristics of the pixels of thesensitivity mosaic image based on the sensitivity mosaic patterninformation to produce the restoration image.

The color interpolation means may include extraction means forextracting those of the pixels which have the same sensitivitycharacteristic from the color and sensitivity mosaic image, all colorcomponent interpolation means for interpolating all of the colorcomponents of the pixels extracted by the extraction means, andsynthesis means for synthesizing those of the pixels having all of thecolor components interpolated by the all color component interpolationmeans which have the same color component and have the differentsensitivity characteristics to produce the sensitivity mosaic image.

The third image processing apparatus of the present invention mayfurther include image pickup means for picking up an image of a subjectto produce the color and sensitivity mosaic image.

A third image processing method of the present invention ischaracterized in that it includes a sensitivity characteristicuniformization step of uniformizing the sensitivity characteristics ofthe pixels based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image, and a color interpolation stepof interpolating color components of the pixels based on color mosaicpattern information representative of an arrangement of the colorcomponents of the color and sensitivity mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce a colormosaic image, and the processing of the color interpolation step mayinterpolate the color components of the pixels of the color mosaic imagebased on the color mosaic pattern information to produce the restorationimage.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image without changing the kinds of the colorcomponents of the pixels of the color and sensitivity mosaic image basedon the sensitivity mosaic pattern information and the color mosaicpattern information to produce the color mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce thecolor mosaic image and update the color mosaic pattern information.

The sensitivity uniformization step may include a compensation step ofcompensating for the color components of the pixels of the color andsensitivity mosaic image based on the sensitivity mosaic patterninformation, a discrimination step of discriminating the validity of thecolor components of the pixels of the color and sensitivity mosaic imagebased on the sensitivity mosaic pattern information, and a modificationstep of modifying the color components of the pixels compensated for bythe processing of the compensation step through an interpolation processin response to a result of the processing at the discrimination step.

The sensitivity uniformization step may include a calculation step ofcalculating estimated pixel values of the color and sensitivity mosaicimage based on the sensitivity mosaic pattern information, and acorrection step of correcting the estimated pixel values calculated bythe processing of the calculation step.

The processing of the color interpolation step may interpolate all ofthe color components of the pixels of the color and sensitivity mosaicimage without changing the sensitivity characteristics of the pixelsbased on the sensitivity mosaic pattern information and the color mosaicpattern information to produce a sensitivity mosaic image of the colorcomponents, and the processing of the sensitivity characteristicuniformization step may uniformize the sensitivity characteristics ofthe pixels of the sensitivity mosaic image based on the sensitivitymosaic pattern information to produce the restoration image.

The color interpolation step may include an extraction step ofextracting those of the pixels which have the same sensitivitycharacteristic from the color and sensitivity mosaic image, an all colorcomponent interpolation step of interpolating all of the colorcomponents of the pixels extracted by the processing of the extractionstep, and a synthesis step of synthesizing those of the pixels havingall of the color components interpolated by the processing of the allcolor component interpolation step which have the same color componentand have the different sensitivity characteristics to produce thesensitivity mosaic image.

The third image processing method of the present invention may furtherinclude an image pickup step of picking up an image of a subject toproduce the color and sensitivity mosaic image.

A program of a third recording medium of the present invention ischaracterized in that the program includes a sensitivity characteristicuniformization step of uniformizing the sensitivity characteristics ofthe pixels based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image, and a color interpolation stepof interpolating color components of the pixels based on color mosaicpattern information representative of an arrangement of the colorcomponents of the color and sensitivity mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce a colormosaic image, and the processing of the color interpolation step mayinterpolate the color components of the pixels of the color mosaic imagebased on the color mosaic pattern information to produce the restorationimage.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image without changing the kinds of the colorcomponents of the pixels of the color and sensitivity mosaic image basedon the sensitivity mosaic pattern information and the color mosaicpattern information to produce the color mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce thecolor mosaic image and update the color mosaic pattern information.

The sensitivity uniformization step may include a compensation step ofcompensating for the color components of the pixels of the color andsensitivity mosaic image based on the sensitivity mosaic patterninformation, a discrimination step of discriminating the validity of thecolor components of the pixels of the color and sensitivity mosaic imagebased on the sensitivity mosaic pattern information, and a modificationstep of modifying the color components of the pixels compensated for bythe processing of the compensation step through an interpolation processin response to a result of the processing at the discrimination step.

The sensitivity uniformization step may include a calculation step ofcalculating estimated pixel values of the color and sensitivity mosaicimage based on the sensitivity mosaic pattern information, and acorrection step of correcting the estimated pixel values calculated bythe processing of the calculation step.

The processing of the color interpolation step may interpolate all ofthe color components of the pixels of the color and sensitivity mosaicimage without changing the sensitivity characteristics of the pixelsbased on the sensitivity mosaic pattern information and the color mosaicpattern information to produce a sensitivity mosaic image of the colorcomponents, and the processing of the sensitivity characteristicuniformization step may uniformize the sensitivity characteristics ofthe pixels of the sensitivity mosaic image based on the sensitivitymosaic pattern information to produce the restoration image.

The color interpolation step may include an extraction step ofextracting those of the pixels which have the same sensitivitycharacteristic from the color and sensitivity mosaic image, an all colorcomponent interpolation step of interpolating all of the colorcomponents of the pixels extracted by the processing of the extractionstep, and a synthesis step of synthesizing those of the pixels havingall of the color components interpolated by the processing of the allcolor component interpolation step which have the same color componentand have the different sensitivity characteristics to produce thesensitivity mosaic image.

The program of the third recording medium of the present invention mayfurther include an image pickup controlling step of controlling aprocess of picking up an image of a subject to produce the color andsensitivity mosaic image.

A third program of the present invention is characterized in that itcauses a computer to execute a sensitivity characteristic uniformizationstep of uniformizing the sensitivity characteristics of the pixels basedon sensitivity mosaic pattern information representative of anarrangement of the sensitivity characteristics of the color andsensitivity mosaic image, and a color interpolation step ofinterpolating color components of the pixels based on color mosaicpattern information representative of an arrangement of the colorcomponents of the color and sensitivity mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce a colormosaic image, and the processing of the color interpolation step mayinterpolate the color components of the pixels of the color mosaic imagebased on the color mosaic pattern information to produce the restorationimage.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image without changing the kinds of the colorcomponents of the pixels of the color and sensitivity mosaic image basedon the sensitivity mosaic pattern information and the color mosaicpattern information to produce the color mosaic image.

The processing of the sensitivity characteristic uniformization step mayuniformize the sensitivity characteristics of the pixels of the colorand sensitivity mosaic image based on the sensitivity mosaic patterninformation and the color mosaic pattern information to produce thecolor mosaic image and update the color mosaic pattern information.

The sensitivity uniformization step may include a compensation step ofcompensating for the color components of the pixels of the color andsensitivity mosaic image based on the sensitivity mosaic patterninformation, a discrimination step of discriminating the validity of thecolor components of the pixels of the color and sensitivity mosaic imagebased on the sensitivity mosaic pattern information, and a modificationstep of modifying the color components of the pixels compensated for bythe processing of the compensation step through an interpolation processin response to a result of the processing at the discrimination step.

The sensitivity uniformization step may include a calculation step ofcalculating estimated pixel values of the color and sensitivity mosaicimage based on the sensitivity mosaic pattern information, and acorrection step of correcting the estimated pixel values calculated bythe processing of the calculation step.

The processing of the color interpolation step may interpolate all ofthe color components of the pixels of the color and sensitivity mosaicimage without changing the sensitivity characteristics of the pixelsbased on the sensitivity mosaic pattern information and the color mosaicpattern information to produce a sensitivity mosaic image of the colorcomponents, and the processing of the sensitivity characteristicuniformization step may uniformize the sensitivity characteristics ofthe pixels of the sensitivity mosaic image based on the sensitivitymosaic pattern information to produce the restoration image.

The color interpolation step may include an extraction step ofextracting those of the pixels which have the same sensitivitycharacteristic from the color and sensitivity mosaic image, an all colorcomponent interpolation step of interpolating all of the colorcomponents of the pixels extracted by the processing of the extractionstep, and a synthesis step of synthesizing those of the pixels havingall of the color components interpolated by the processing of the allcolor component interpolation step which have the same color componentand have the different sensitivity characteristics to produce thesensitivity mosaic image.

The third program of the present invention may further include an imagepickup controlling step of controlling a process of picking up an imageof a subject to produce the color and sensitivity mosaic image.

A fourth image processing apparatus of the present invention ischaracterized in that it includes luminance image production means forproducing a luminance image corresponding to the color and sensitivitymosaic image based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image and color mosaic patterninformation representative of an arrangement of the color components ofthe color and sensitivity mosaic image, and a plurality of monochromaticimage production means each for producing a monochromatic imagecorresponding to the color and sensitivity mosaic image based on thesensitivity mosaic pattern information, the color mosaic patterninformation and the luminance image.

The luminance image production means may include a plurality ofestimation means each for calculating an estimated value of a colorcomponent corresponding to each of the pixels of the color andsensitivity mosaic image, and luminance candidate value calculationmeans for calculating a luminance candidate value corresponding to eachof the pixels of the color and sensitivity mosaic image using aplurality of the estimated values calculated individually by theplurality of estimation means.

Each of the estimation means may calculate a plurality of estimatedvalue candidates individually corresponding to the plurality ofsensitivity characteristics, add the plurality of estimated valuecandidates and compensate for the non-linearity of the sensitivitycharacteristic appearing with the sum of the plurality of estimatedvalue candidates.

The luminance image production means may further include noise removalmeans for removing noise components of the luminance candidate value toproduce a luminance value.

Each of the monochromatic image production means may includemonochromatic image candidate production means for producing amonochromatic image candidate corresponding to the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information, and modification means for modifyingthe monochromatic image candidate based on the luminance image toproduce the monochromatic image.

The monochromatic image candidate production means may calculate aplurality of monochromatic candidate values individually correspondingto the plurality of sensitivity characteristics, add the plurality ofmonochromatic candidate values and compensate for the non-linearity ofthe sensitivity characteristic appearing with the sum of the pluralityof monochromatic candidate values to calculate pixel values of themonochromatic image candidate to produce the monochromatic imagecandidate.

The monochromatic image candidate production means may use a directionselective smoothing process to produce the monochromatic image candidatecorresponding to the color and sensitivity mosaic image.

The fourth image processing apparatus of the present invention mayfurther include image pickup means for picking up an image of a subjectto produce the color and sensitivity mosaic image.

A fourth image processing method of the present invention ischaracterized in that it includes a luminance image production step ofproducing a luminance image corresponding to the color and sensitivitymosaic image based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image and color mosaic patterninformation representative of an arrangement of the color components ofthe color and sensitivity mosaic image, and a plurality of monochromaticimage production steps each of producing a monochromatic imagecorresponding to the color and sensitivity mosaic image based on thesensitivity mosaic pattern information, the color mosaic patterninformation and the luminance image.

The luminance image production step may include a plurality ofestimation steps each of calculating an estimated value of a colorcomponent corresponding to each of the pixels of the color andsensitivity mosaic image, and a luminance candidate value calculationstep of calculating a luminance candidate value corresponding to each ofthe pixels of the color and sensitivity mosaic image using a pluralityof the estimated values calculated individually by the processing of theplurality of estimation steps.

The processing of each of the estimation steps may calculate a pluralityof estimated value candidates individually corresponding to theplurality of sensitivity characteristics, add the plurality of estimatedvalue candidates and compensate for the non-linearity of the sensitivitycharacteristic appearing with the sum of the plurality of estimatedvalue candidates.

The luminance image production step may further include a noise removalstep of removing noise components of the luminance candidate value toproduce a luminance value.

Each of the monochromatic image production steps may include amonochromatic image candidate production step of producing amonochromatic image candidate corresponding to the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information, and a modification step of modifyingthe monochromatic image candidate based on the luminance image toproduce the monochromatic image.

The processing of the monochromatic image candidate production step maycalculate a plurality of monochromatic candidate values individuallycorresponding to the plurality of sensitivity characteristics, add theplurality of monochromatic candidate values and compensate for thenon-linearity of the sensitivity characteristic appearing with the sumof the plurality of monochromatic candidate values to calculate pixelvalues of the monochromatic image candidate to produce the monochromaticimage candidate.

The processing of the monochromatic image candidate production step mayuse a direction selective smoothing process to produce the monochromaticimage candidate corresponding to the color and sensitivity mosaic image.

The fourth image processing method of the present invention may furtherinclude an image pickup step of picking up an image of a subject toproduce the color and sensitivity mosaic image.

A program of a fourth recording medium of the present invention ischaracterized in that the program includes a luminance image productionstep of producing a luminance image corresponding to the color andsensitivity mosaic image based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image and color mosaic patterninformation representative of an arrangement of the color components ofthe color and sensitivity mosaic image, and a plurality of monochromaticimage production steps each of producing a monochromatic imagecorresponding to the color and sensitivity mosaic image based on thesensitivity mosaic pattern information, the color mosaic patterninformation and the luminance image.

The luminance image production step may include a plurality ofestimation steps each of calculating an estimated value of a colorcomponent corresponding to each of the pixels of the color andsensitivity mosaic image, and a luminance candidate value calculationstep of calculating a luminance candidate value corresponding to each ofthe pixels of the color and sensitivity mosaic image using a pluralityof the estimated values calculated individually by the processing of theplurality of estimation steps.

The processing of each of the estimation steps may calculate a pluralityof estimated value candidates individually corresponding to theplurality of sensitivity characteristics, add the plurality of estimatedvalue candidates and compensate for the non-linearity of the sensitivitycharacteristic appearing with the sum of the plurality of estimatedvalue candidates.

The luminance image production step may further include a noise removalstep of removing noise components of the luminance candidate value toproduce a luminance value.

Each of the monochromatic image production steps may include amonochromatic image candidate production step of producing amonochromatic image candidate corresponding to the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information, and a modification step of modifyingthe monochromatic image candidate based on the luminance image toproduce the monochromatic image.

The processing of the monochromatic image candidate production step maycalculate a plurality of monochromatic candidate values individuallycorresponding to the plurality of sensitivity characteristics, add theplurality of monochromatic candidate values and compensate for thenon-linearity of the sensitivity characteristic appearing with the sumof the plurality of monochromatic candidate values to calculate pixelvalues of the monochromatic image candidate to produce the monochromaticimage candidate.

The processing of the monochromatic image candidate production step mayuse a direction selective smoothing process to produce the monochromaticimage candidate corresponding to the color and sensitivity mosaic image.

The program of the fourth recording medium of the present invention mayfurther include an image pickup controlling step of controlling aprocess of picking up an image of a subject to produce the color andsensitivity mosaic image.

A fourth program of the present invention is characterized in that itcauses a computer to execute a luminance image production step ofproducing a luminance image corresponding to the color and sensitivitymosaic image based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image and color mosaic patterninformation representative of an arrangement of the color components ofthe color and sensitivity mosaic image, and a plurality of monochromaticimage production steps each of producing a monochromatic imagecorresponding to the color and sensitivity mosaic image based on thesensitivity mosaic pattern information, the color mosaic patterninformation and the luminance image.

The luminance image production step may include a plurality ofestimation steps each of calculating an estimated value of a colorcomponent corresponding to each of the pixels of the color andsensitivity mosaic image, and a luminance candidate value calculationstep of calculating a luminance candidate value corresponding to each ofthe pixels of the color and sensitivity mosaic image using a pluralityof the estimated values calculated individually by the processing of theplurality of estimation steps.

The processing of each of the estimation steps may calculate a pluralityof estimated value candidates individually corresponding to theplurality of sensitivity characteristics, add the plurality of estimatedvalue candidates and compensate for the non-linearity of the sensitivitycharacteristic appearing with the sum of the plurality of estimatedvalue candidates.

The luminance image production step may further include a noise removalstep of removing noise components of the luminance candidate value toproduce a luminance value.

Each of the monochromatic image production steps may include amonochromatic image candidate production step of producing amonochromatic image candidate corresponding to the color and sensitivitymosaic image based on the sensitivity mosaic pattern information and thecolor mosaic pattern information, and a modification step of modifyingthe monochromatic image candidate based on the luminance image toproduce the monochromatic image.

The processing of the monochromatic image candidate production step maycalculate a plurality of monochromatic candidate values individuallycorresponding to the plurality of sensitivity characteristics, add theplurality of monochromatic candidate values and compensate for thenon-linearity of the sensitivity characteristic appearing with the sumof the plurality of monochromatic candidate values to calculate pixelvalues of the monochromatic image candidate to produce the monochromaticimage candidate.

The processing of the monochromatic image candidate production step mayuse a direction selective smoothing process to produce the monochromaticimage candidate corresponding to the color and sensitivity mosaic image.

The fourth program of the present invention may further include an imagepickup controlling step of controlling a process of picking up an imageof a subject to produce the color and sensitivity mosaic image.

In the first image processing apparatus and method as well as program ofthe present invention, based on a color and sensitivity mosaic imagewherein each of a plurality of pixels has one of a plurality of colorcomponents and one of a plurality of sensitivity characteristics withrespect to the intensity of light and a plurality of ones of the pixelswhich have the same color component and the same sensitivitycharacteristic are arranged in a grating-like arrangement and besides aplurality of ones of the pixels which have the same color componentirrespective of the sensitivity characteristic are arranged in agrating-like arrangement, a restoration image wherein the sensitivitiesof the pixels are uniformized and each of the pixels has all of theplurality of color components is restored.

In the second image processing apparatus and method as well as programof the present invention, based on a color and sensitivity mosaic imagewherein each of a plurality of pixels has one of a plurality of colorcomponents and one of a plurality of sensitivity characteristics withrespect to the intensity of light and a plurality of ones of the pixelswhich have the same color component and the same sensitivitycharacteristic are arranged in a grating-like arrangement and besides aplurality of ones of the pixels which have the same sensitivitycharacteristic irrespective of the color component are arranged in agrating-like arrangement such that totaling 5 pixels including anarbitrary pixel and four pixels neighboring upwardly, downwardly,leftwardly and rightwardly of the arbitrary pixel include all of thecolor components, a restoration image wherein the sensitivities of thepixels are uniformized and each of the pixels has all of the pluralityof color components is restored.

In the third image processing apparatus and method as well as program ofthe present invention, the sensitivity characteristics of the pixels areuniformized based on sensitivity mosaic pattern informationrepresentative of an arrangement of the sensitivity characteristics ofthe color and sensitivity mosaic image, and color components of thepixels are interpolated based on color mosaic pattern informationrepresentative of an arrangement of the color components of the colorand sensitivity mosaic image.

In the fourth image processing apparatus and method as well as programof the present invention, a luminance image corresponding to the colorand sensitivity mosaic image is produced based on sensitivity mosaicpattern information representative of an arrangement of the sensitivitycharacteristics of the color and sensitivity mosaic image and colormosaic pattern information representative of an arrangement of the colorcomponents of the color and sensitivity mosaic image, and amonochromatic image corresponding to the color and sensitivity mosaicimage is produced based on the sensitivity mosaic pattern information,the color mosaic pattern information and the luminance image.

Additional features and advantages are described herein, and will beapparent from the following Detailed Description and the figures.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram showing an example of a configuration of adigital still camera to which the present invention is applied;

FIG. 2 is a view illustrating general operation of the digital stillcamera;

FIG. 3 is a view showing an example of a subject;

FIG. 4 is a view showing an example of a color and sensitivity mosaicimage corresponding to the example of FIG. 3;

FIG. 5 is a view showing a color and sensitivity mosaic pattern P1;

FIG. 6 is a view showing a color and sensitivity mosaic pattern P2;

FIG. 7 is a view showing a color and sensitivity mosaic pattern P3;

FIG. 8 is a view showing a color and sensitivity mosaic pattern P4;

FIG. 9 is a view showing a color and sensitivity mosaic pattern P5;

FIG. 10 is a view showing a color and sensitivity mosaic pattern P6;

FIG. 11 is a view showing a color and sensitivity mosaic pattern P7;

FIG. 12 is a view showing a color and sensitivity mosaic pattern P8;

FIG. 13 is a view showing a color and sensitivity mosaic pattern P9;

FIG. 14 is a view showing a color and sensitivity mosaic pattern P10;

FIG. 15 is a view showing a color and sensitivity mosaic pattern P11;

FIG. 16 is a view showing a color and sensitivity mosaic pattern P12;

FIG. 17 is a view showing a color and sensitivity mosaic pattern P13;

FIG. 18 is a view showing a color and sensitivity mosaic pattern P14;

FIG. 19 is a view showing a cross section of a light receiving elementof a CCD image sensor 4;

FIG. 20 is a view illustrating a method for optically implementing amosaic arrangement of sensitivity;

FIG. 21 is a view illustrating another method for optically implementinga mosaic arrangement of sensitivity;

FIG. 22 is a view illustrating a further method for opticallyimplementing a mosaic pattern of sensitivity;

FIG. 23 is a view illustrating a first method for electronicallyimplementing a mosaic pattern of sensitivity;

FIG. 24 is a view illustrating a second method for electronicallyimplementing a mosaic pattern of sensitivity;

FIG. 25 is a schematic view showing an OR-type electrode structure;

FIG. 26 is a sectional view showing a cross section of the OR-typeelectrode structure;

FIG. 27 is a schematic view showing an AND-type electrode structure;

FIG. 28 is a view showing a combination of the OR-type electrodestructure and the AND-type electrode structure for implementing thecolor and sensitivity mosaic pattern P1;

FIG. 29 is a view showing a combination of the OR-type electrodestructure and the AND-type electrode structure for implementing thecolor and sensitivity mosaic pattern P2;

FIG. 30 is a view showing a combination of the OR-type electrodestructure and the AND-type electrode structure for implementing thecolor and sensitivity mosaic pattern P3;

FIG. 31 is a view showing a combination of the OR-type electrodestructure and the AND-type electrode structure for implementing thecolor and sensitivity mosaic pattern P4;

FIG. 32 is a view showing a combination of the OR-type electrodestructure and the AND-type electrode structure for implementing thecolor and sensitivity mosaic pattern P5;

FIG. 33 is a view illustrating a definition of position coordinates of apixel;

FIG. 34 is a view illustrating an outline of a first demosaic process;

FIG. 35 is a graph illustrating an outline of a first sensitivityuniformization process in the first demosaic process;

FIG. 36 is a graph illustrating an outline of the first sensitivityuniformization process in the first demosaic process;

FIG. 37 is a graph illustrating an outline of the first sensitivityuniformization process in the first demosaic process;

FIG. 38 is a graph illustrating an outline of a second sensitivityuniformization process in the first demosaic process;

FIG. 39 is a graph illustrating an outline of the second sensitivityuniformization process in the first demosaic process;

FIG. 40 is a view illustrating an outline of a second demosaic process;

FIG. 41 is a graph illustrating an outline of a first sensitivityuniformization process in the second demosaic process;

FIG. 42 is a graph illustrating an outline of the first sensitivityuniformization process in the second demosaic process;

FIG. 43 is a graph illustrating an outline of a second sensitivityuniformization process in the second demosaic process;

FIG. 44 is a graph illustrating an outline of the second sensitivityuniformization process in the second demosaic process;

FIG. 45 is a block diagram showing a first example of a configuration ofan image processing section 7;

FIG. 46 is a block diagram showing a first example of a configuration ofa sensitivity uniformization section 51;

FIG. 47 is a block diagram showing an example of a configuration of acolor interpolation section 52;

FIG. 48 is a block diagram showing an example of a configuration of acolor difference image production section 72;

FIG. 49 is a block diagram showing an example of a configuration of aluminance image production section 74;

FIG. 50 is a flow chart illustrating the first demosaic process by thefirst example of the configuration of the image processing section 7;

FIG. 51 is a flow chart illustrating the first sensitivityuniformization process by the first example of the configuration of thesensitivity uniformization section 51;

FIG. 52 is a flow chart illustrating a sensitivity compensation processat step S11;

FIG. 53 is a flow chart illustrating a validity discrimination processat step S12;

FIG. 54 is a flow chart illustrating a missing interpolation process atstep S13;

FIG. 55 is a flow chart illustrating a color interpolation process atstep S2;

FIG. 56 is a flow chart illustrating a first color difference imageproduction process at step S52;

FIG. 57 is flow chart illustrating a luminance image production processat step S53;

FIG. 58 is a flow chart illustrating a color space conversion process atstep S54;

FIG. 59 is a block diagram showing a second example of a configurationof the sensitivity uniformization section 51;

FIG. 60 is a flow chart illustrating a second sensitivity uniformizationprocess by the second example of the configuration of the sensitivityuniformization section 51;

FIG. 61 is a flow chart illustrating an interpolation process at stepS103;

FIG. 62 is a flow chart illustrating a second color difference imageproduction process;

FIG. 63 is a flow chart illustrating an image gradient vector arithmeticoperation process at step S123;

FIG. 64 is a block diagram showing a second example of a configurationof the image processing section 7;

FIG. 65 is a block diagram showing a first example of a configuration ofa sensitivity uniformization section 111;

FIG. 66 is a flow chart illustrating a missing interpolation process bya missing interpolation section 124;

FIG. 67 is a block diagram showing a second example of a configurationof the sensitivity uniformization section 111;

FIG. 68 is a flow chart illustrating the second sensitivityuniformization process in the second demosaic process by the secondexample of the configuration of the sensitivity uniformization section111;

FIG. 69 is a flow chart illustrating an interpolation colordetermination process at step S163;

FIG. 70 is a view illustrating an outline of a third demosaic process;

FIG. 71 is a view illustrating an outline of a by-sensitivity-basiscolor interpolation process in the third demosaic process;

FIG. 72 is a view illustrating an outline of the by-sensitivity-basiscolor interpolation process in the third demosaic process;

FIG. 73 is a block diagram showing a third example of a configuration ofthe image processing section 7;

FIG. 74 is a block diagram showing an example of a configuration of aby-sensitivity-basis color interpolation section 151;

FIG. 75 is a block diagram showing an example of a configuration of asensitivity uniformization section 152;

FIG. 76 is a flow chart illustrating the third demosaic process by thethird example of the configuration of the image processing section 7;

FIG. 77 is a flow chart illustrating the by-sensitivity-basis colorinterpolation process at step S181;

FIG. 78 is a view illustrating an extraction process at step S193;

FIG. 79 is a view illustrating the extraction process at step S193;

FIG. 80 is a flow chart illustrating a sensitivity uniformizationprocess at step S182;

FIG. 81 is a view showing an example of a filter coefficient used in alocal sum calculation process at step S203;

FIG. 82 is a block diagram showing a fourteen example of a configurationof the image processing section 7;

FIG. 83 is a block diagram showing a first example of a configuration ofa luminance image production section 181;

FIG. 84 is a block diagram showing an example of a configuration of amonochromatic image production section 182;

FIG. 85 is a flow chart illustrating a fourth demosaic process by thefourth example of the configuration of the image processing section 7;

FIG. 86 is a flow chart illustrating a luminance image productionprocess by the luminance image production section 181;

FIG. 87 is a flow chart illustrating an R component estimation processby an estimation section 191;

FIG. 88 is a view showing an example of interpolation filtercoefficients for R/B components;

FIG. 89 is a view showing interpolation filter coefficients for a Gcomponent;

FIG. 90 is a view illustrating a synthetic sensitivity compensation LUT;

FIG. 91 is a view illustrating another synthetic sensitivitycompensation LUT;

FIG. 92 is a view illustrating a further synthetic sensitivitycompensation LUT;

FIG. 93 is a flow chart illustrating a noise removal process by a noiseremoval section 198;

FIG. 94 is a flow chart illustrating a direction selective smoothingprocess by the noise removal section 198;

FIG. 95 is a flow chart illustrating a monochromatic image productionprocess by the monochromatic image production section 182;

FIG. 96 is a flow chart illustrating a ratio value calculation processby a ratio value calculation section 202;

FIG. 97 is a view illustrating an example of smoothing filtercoefficients;

FIG. 98 is a block diagram showing a second example of a configurationof the luminance image production section 181;

FIG. 99 is a flow chart illustrating an estimation process of RGBcomponents by an estimation section 211;

FIG. 100 is a view showing an arrangement of pixels used in anestimation pixel value C0 interpolation process;

FIG. 101 is a flow chart illustrating the estimation pixel value C0interpolation process;

FIG. 102 is a view showing an arrangement of pixels used in anestimation pixel value C1 interpolation process;

FIG. 103 is a flow chart illustrating the estimation pixel value C1interpolation process;

FIG. 104A is a view showing an arrangement of pixels used in anestimation pixel value C2 interpolation process;

FIG. 104B is a view showing another arrangement of pixels used in theestimation pixel value C2 interpolation process;

FIG. 105 is a flow chart illustrating the estimation pixel value C2interpolation process;

FIG. 106 is a view showing an arrangement of pixels used in anestimation pixel value C3 interpolation process;

FIG. 107 is a flow chart illustrating the estimation pixel value C3interpolation process;

FIG. 108 is a flow chart illustrating an R candidate image productionprocess by an interpolation section 201-R;

FIG. 109 is a flow chart illustrating a B candidate image productionprocess by an interpolation section 201-B;

FIG. 110 is a flow chart illustrating a G candidate image productionprocess by an interpolation section 201-G; and

FIG. 111 is a block diagram showing a fifth example of a configurationof the image processing section 7.

DETAILED DESCRIPTION

FIG. 1 shows an example of a configuration of a digital still camerawhich is an embodiment of the present invention. The digital stillcamera is roughly composed of an optical system, a signal processingsystem, a recording system, a display system and a control system.

The optical system includes a lens 1 for condensing an optical image ofa subject, an iris 2 for adjusting the amount of light of the opticalimage, and a CCD image sensor 4 for photo-electrically converting thecondensed optical image into an electric signal of a wide dynamic range.

The signal processing system includes a correlation double samplingcircuit (CDS) 5 for sampling an electric signal from the CCD imagesensor 4 to reduce noise of the electric signal, an A/D converter 6 forconverting an analog signal outputted from the correlation doublesampling circuit 5 into a digital signal, and an image processingsection 7 for performing a predetermined image process for the digitalsignal inputted thereto from the A/D converter 6. It is to be noted thatdetails of the process executed by the image processing section 7 arehereinafter described.

The recording system includes a CODEC (Compression/Decompression) 8 forcoding and recording an image signal processed by the image processingsection 7 into a memory 9 and reading out, decoding and supplying theimage signal to the image processing section 7, and the memory 9 forstoring an image signal.

The display system includes a D/A converter 10 for converting an imagesignal processed by the image processing section 7 into an analogsignal, a video encoder 11 for encoding the analog image signal into avideo signal of the format compatible with a display unit 12 in thefollowing stage, and a display unit 12 formed from an LCD (LiquidCrystal Display) unit or the like for displaying an image correspondingto the video signal inputted thereto so that it functions as aviewfinder.

The control system includes a timing generator (TG) 3 for controllingoperation timings of the components from the CCD image sensor 4 to theimage processing section 7, an operation inputting section 13 forallowing the user to input a shutter timing and other commands, and acontrol section 14 including a CPU (Central Processing Unit) and soforth for controlling a drive 15 to read out a controlling programstored on a magnetic disc 16, an optical disc 17, a magneto-optical disc18 or a semiconductor memory 19 and controlling the entire digital stillcamera based on the controlling program read out, a command from theuser inputted from the operation inputting section 13 and so forth.

In the digital still camera, an optical image (incoming light) of asubject is introduced into the CCD image sensor 4 through the lens 1 andthe iris 2, and it is photo-electrically converted by the CCD imagesensor 4. The resulting electric signal is subject to removal of noiseby the correlation double sampling circuit 5 and is then converted intoa digital signal by the A/D converter 6, whereafter it is temporarilystored into an image memory built in the image processing section 7.

It is to be noted that, in an ordinary state, an image signal isincessantly overwritten at a fixed frame rate into the image memorybuilt in the image processing section 7 under the control of the timinggenerator 3 for the signal processing system. The image signal of theimage memory built in the image processing section 7 is converted intoan analog signal by the D/A converter 10 and further converted into avideo signal by the video encoder 11, and a corresponding image isdisplayed on the display unit 12.

The display unit 12 further has a function as a viewfinder of thedigital still camera. When the user depresses a shutter button includedin the operation inputting section 13, the control section 14 controlsthe timing generator 3 so that the signal processing system fetches animage signal immediately after the shutter button is depressed andthereafter inhibits an image signal from being overwritten into theimage memory of the image processing section 7. Thereafter, the imagedata written in the image memory of the image processing section 7 arecoded by the CODEC 8 and recorded into the memory 9. Fetching of imagedata of one frame is completed by such operation of the digital stillcamera as described above.

Subsequently, an outline of operation of the digital still camera isdescribed with reference to FIG. 2. The digital still camera picks up animage of a subject with a color and a sensitivity, which are differentfor each pixel, through an image pickup process of the optical systemincluding the CCD image sensor 4 as a principal component to obtain animage wherein colors and sensitivities are distributed like a mosaic(such an image as just described is hereinafter referred to as color andsensitivity mosaic image, whose details are hereinafter described).Thereafter, the image obtained by the image pickup process is convertedinto an image wherein each pixel has all color components and the pixelshave a uniform sensitivity by the signal processing system whichincludes the image processing section 7 as a principal component. In thefollowing description, the process of the signal processing systemincluding the image processing section 7 as a principal component forconverting a color and sensitivity mosaic image into an image whereineach pixel has all color components and the pixels have a uniformsensitivity is referred to as demosaic process.

For example, if an image of such a subject as shown in FIG. 3 is pickedup, then such a color and sensitivity mosaic image as shown in FIG. 4 isobtained through the image pickup process and is converted into an imagewherein each pixel has all color components and the pixels have auniform sensitivity through the image process. In particular, theoriginal colors of the subject shown in FIG. 3 are restored from thecolor and sensitivity mosaic image shown in FIG. 4.

Arrangement patterns (hereinafter referred to as color and sensitivitymosaic patterns) P1 to P14 of color components and sensitivities ofpixels which compose a color and sensitivity mosaic image are shown inFIGS. 5 to 18, respectively. It is to be noted that, as a combination ofcolors which form a color and sensitivity mosaic pattern, a combinationof three colors of R (red), G (green) and B (blue) and anothercombination of four colors of Y (yellow), M (magenta), C (cyan) and G(green) are available. As stages of the sensitivity, two stages of S0and S1, three stages which additionally include a sensitivity S2 andfour stages which additionally include a further sensitivity S3 areavailable. It is to be noted that, in FIGS. 5 to 18, each squarecorresponds to one pixel, and an alphabetical letter represents thecolor of the pixel and a numeral as a subscript to the alphabeticalletter represents the sensitivity of the pixel. For example, a pixeldenoted by G₀ represents that the color thereof is G (green) and thesensitivity thereof is S0. Further, it is assumed that, as regards thesensitivity, the higher the value, the higher the sensitivity.

The color and sensitivity mosaic patterns P1 to P14 can be classifiedbased on the first to fourth characteristics described below.

The first characteristic is that, where attention is paid to thosepixels which have the same color and the same sensitivity, they arearranged like a grating, and where attention is paid to those pixelswhich have the same color irrespective of the sensitivity, they arearranged like a grating. The first characteristic is described withreference to the color and sensitivity mosaic pattern P1 shown in FIG.5.

In the color and sensitivity mosaic pattern P1 of FIG. 5, whereattention is paid to those pixels which have the color R irrespective ofthe sensitivity, as can be seen apparently if the figure is viewed in astate rotated by 45 degrees in the clockwise direction, they arearranged like a grating wherein they are spaced from each other by2^(1/2) in the horizontal direction and by 2^(3/2) in the verticaldirection. Further, where attention is paid to those pixels which havethe color B irrespective of the sensitivity, also they are arranged likea grating wherein they are spaced from each other by 2^(1/2) in thehorizontal direction and by 2^(3/2) in the vertical direction. Further,where attention is paid to those pixels which have the color Girrespective of the sensitivity, also they are arranged like a gratingwherein they are spaced from each other by 2^(1/2) both in thehorizontal direction and in the vertical direction.

In addition to the color and sensitivity mosaic pattern P1 shown in FIG.5, the color and sensitivity mosaic patterns P2, P4, P6, P8, P9, P10,P11 and P13 have the first characteristic.

The second characteristic is that, where attention is paid to thosepixels which have the same color and the same sensitivity, they arearranged like a grating, and where attention is paid to those pixelswhich have the same sensitivity irrespective of the color, they arearranged like a grating, and besides, where attention is paid to anarbitrary pixel, all of colors included in the color and sensitivitymosaic pattern are included in colors which totaling five pixelsincluding the pixel and four pixels positioned upwardly, downwardly,leftwardly and rightwardly of the pixel have.

In addition to the color and sensitivity mosaic pattern P3 shown in FIG.7, the color and sensitivity mosaic patterns P5, P7, P8, P9, P12 and P14have the second characteristic.

The third characteristic is that the color and sensitivity mosaicpattern has the first characteristic and uses three different colors andthe pixels of the colors are arranged in a Bayer arrangement. The thirdcharacteristic is described with reference to the color and sensitivitymosaic pattern P2 shown in FIG. 6.

Where attention is paid to those pixels of the color and sensitivitymosaic pattern P2 of FIG. 6 which have the color G irrespective of thesensitivity, they are arranged alternately in a checkered pattern. Whereattention is paid to those pixels which have the color R irrespective ofthe sensitivity, they are arranged on every other line. Further, alsowhere attention is paid to those pixels whose color is B irrespective ofthe sensitivity, they are arranged on every other line similarly.Accordingly, the pattern P2 has a Bayer arrangement where attention ispaid only to the colors of the pixels.

It is to be noted that, in addition to the color and sensitivity mosaicpattern P2 of FIG. 6, the color and sensitivity mosaic patterns P10 andP11 have the third characteristic.

The fourth characteristic is that the color and sensitivity mosaicpattern has the second characteristic and further, where attention ispaid to those pixels which have the same sensitivity, the arrangement ofthem is a Bayer arrangement. The fourth characteristic is described withreference to the color and sensitivity mosaic pattern P3 shown in FIG.7.

Where attention is paid only to those pixels in the color andsensitivity mosaic pattern P3 shown in FIG. 7 which have the sensitivityS0, as can be seen apparently if the figure is viewed obliquely in astate inclined by 45 degrees, they are arranged in a spaced relationshipby a distance of 2^(1/2) and in a Bayer arrangement. Also whereattention is paid to those pixels which have the sensitivity S1, theyare arranged in a Bayer arrangement similarly.

It is to be noted that, in addition to the color and sensitivity mosaicpattern P3 of FIG. 7, the color and sensitivity mosaic patterns P5 andP12 have the fourth characteristic.

Incidentally, an arrangement of any of the color and sensitivity mosaicpatterns P1 to P14 shown in FIGS. 5 to 18 is hereinafter referred to as“color mosaic arrangement” where attention is paid only to the colors ofthe pixels irrespective of the sensitivity, but is hereinafter referredto as “sensitivity mosaic arrangement” where attention is paid only tothe sensitivities irrespective of the color.

Subsequently, a method of implementing the color and sensitivity mosaicpatterns described above on the CCD image sensor 4 is described.

Of the color and sensitivity mosaic patterns, the color mosaicarrangements are implemented by disposing an on-chip color filter, whichpasses only light of a different color for each pixel, on an upper faceof a light receiving element of the CCD image sensor 4.

Of the color and sensitivity mosaic patterns, the sensitivity mosaicarrangements are implemented by an optical method or an electronicmethod.

A method of optically implementing a sensitivity mosaic arrangement isdescribed. FIG. 19 shows a cross section of a light receiving element ofthe CCD image sensor 4. An on-chip lens 21 is formed on an upper surfaceof the light receiving element. The on-chip lens 21 is disposed so thatit condenses incoming light from an upper portion of the figure on aphoto-diode (PD) 23. An on-chip color filter 22 limits a wavelength bandof the incoming light (passes only a particular wavelength bandtherethrough). The photo-diode 23 is formed in a wafer at a lowerportion of the light receiving element. The photo-diode 23 produceselectric charge in response to the amount of light inputted thereto. Avertical register 26 is formed on the opposite sides of the photo-diode21. A pair of vertical register driving electrodes 25 for driving thevertical register 21 are wired above the vertical register 26.

Since the vertical register 26 is a region for transferring electriccharge produced by the photo-diode 23, the vertical register 26 and thevertical register driving electrodes 25 are shielded from light by ashield 24 so that no electric charge may be produced in the verticalregister 26. The shield 24 is open only above the photo-diode 23 suchthat the incoming light may pass the opening portion until it reachesthe photodiode 23.

The sensitivity of each light receiving element can be varied (theamount of incoming light to the photo-diode 23 can be varied) making useof the CCD image sensor 4 configured in such a manner as describedabove.

For example, the amount of condensed light can be varied depending uponwhether or not the on-chip lens 21 is disposed as seen in FIG. 20.Meanwhile, the light transmission factor can be varied, for example, bydisposing a neutral density filter 31 above (or below) the on-chip colorfilter 22 as seen in FIG. 21. Further, the incoming light amount to thephoto-diode 23 can be varied, for example, by varying the area of theopening portion of the shield 24 as seen in FIG. 22.

Now, two different methods for electronically implementing a mosaicarrangement of sensitivity are described.

For example, a first method of setting two adjacent light receivingelements (first and second light receiving elements) to differentsensitivities by changing the timing of control is described withreference to FIG. 23.

The first stage of FIG. 23 shows an exposure period of the CCD imagesensor 4. The second stage of FIG. 23 shows a timing of a pulse voltagefor instruction of sweeping out of electric charge. The third stage ofFIG. 23 shows a timing at which a control voltage for instruction ofcharge transfer is applied. The fourth stage of FIG. 23 shows a timingof a pulse voltage for instructing a first light receiving element toread out electric charge. The fifth stage of FIG. 23 shows a variationof the electric charge amount accumulated in the first light receivingelement in response to application of the charge sweeping out pulsevoltage and the charge reading out pulse voltage. The sixth stage ofFIG. 23 shows a timing of a pulse voltage for instructing a second lightreceiving element to read out electric charge. The seventh stage of FIG.23 shows a variation of the electric charge amount accumulated in thesecond light receiving element in response to application of the chargesweeping out pulse voltage and the charge reading out pulse voltage.

In the first method of electronically implementing a sensitivity mosaicarrangement, the charge sweeping out pulse voltage is supplied commonlyto the first and second light receiving elements so that, except withinan exposure period, electric charge is swept out (reset) from thephoto-diode 23, but within an exposure period, electric charge is resetonly once at a predetermined timing.

The charge transfer voltage is supplied, except within an exposureperiod, as a waveform voltage for transferring electric charge to thevertical register 26 commonly to the first and second light receivingelements, but is not supplied, within an exposure period, so thattransfer of electric charge from the vertical register 26 may bestopped.

The charge reading out pulse voltage is supplied at different timings tothe light receiving elements. To the first light receiving element, thecharge reading out pulse voltage for the first time is suppliedimmediately before the supplying timing of the charge sweeping outvoltage within an exposure period (second stage of FIG. 23), but thecharge reading out pulse voltage for the second time is suppliedimmediately before the end of the exposure period.

As a result, from the first light receiving element, the accumulatedcharge amount of the first light receiving element is read out into thevertical register 26 at the supplying timings of the charge reading outpulse voltage for the first and second times. It is to be noted that,since transfer of electric charge of the vertical register 26 stopswithin an exposure period, the electric charge amounts read out twiceare added in the vertical register 26 and transferred as data of thesame frame from the vertical register 26 after the end of the exposureperiod.

Meanwhile, to the second light receiving element, the charge reading outpulse voltage is supplied only once immediately before the supplyingtiming of the charge sweeping out pulse voltage within an exposureperiod. As a result, from the second light receiving element, theaccumulated electric charge amount of the second light receiving elementat the only one supplying timing of the charge reading out pulse voltageis read out into the vertical register 26. It is to be noted that, sincetransfer of electric charge of the vertical register 23 stops within anexposure period, the accumulated electric charge read out from thesecond light receiving element is transferred as data of the same frameas that of the accumulated electric charge read out from the first lightreceiving element from the vertical register 26 after the end of theexposure period.

By making the control timings for the first light receiving element andthe second light receiving element different from each other in thismanner, it is possible to set so that the accumulated electric chargeamount read out from the first light receiving element and theaccumulated electric charge amount read out from the second lightreceiving element within the same exposure period, or in other words,the sensitivities, may be different from each other.

Incidentally, the first method of electronically implementing asensitivity mosaic arrangement has a problem in that, depending upon alight receiving element, information of a subject cannot be measuredover an overall region within an exposure period.

Now, a second method of electronically implementing a sensitivity mosaicarrangement is described with reference to FIG. 24. The first to sixthstages of FIG. 24 show, similarly to the first to sixth stages of FIG.23, an exposure period of the CCD image sensor 4, a timing of a pulsevoltage for instruction of sweeping out of electric charge, a timing atwhich a control voltage for instruction of charge transfer is applied, atiming of a pulse voltage for instructing the first light receivingelement to read out electric charge, a variation of the electric chargeamount accumulated in the first light receiving element in response toapplication of the charge sweeping out pulse voltage and the chargereading out pulse voltage, a timing of a pulse voltage for instructingthe second light receiving element to read out electric charge, and avariation of the electric charge amount accumulated in the second lightreceiving element in response to application of the charge sweeping outpulse voltage and the charge reading out pulse voltage.

In the second method of electronically implementing a sensitivity mosaicarrangement, the charge sweeping out pulse voltage and the chargereading out pulse voltage are supplied repetitively by a plural numberof times within an exposure period.

In particular, as regards the charge sweeping out pulse voltage, a setof the charge sweeping out pulse voltage for the first time and thecharge sweeping out pulse voltage for the second time are supplied by aplural number of times commonly to the first and second light receivingelements within an exposure period. As regards the charge reading outpulse voltage, to the first light receiving element, the charge readingout pulse voltage for the first time is supplied, for each set of thecharge sweeping out pulse voltages for the first and second times,immediately before the charge sweeping out pulse voltage for the firsttime, and the charge reading out pulse voltage for the second time issupplied immediately before the charge sweeping out pulse voltage forthe second time. Meanwhile, to the second light receiving element, foreach set of the charge sweeping out pulse voltages, the charge readingout pulse voltage is supplied only once immediately before the chargesweeping out pulse voltage for the first time.

As a result, for each set of the charge sweeping out pulse voltages forthe first and second times, the accumulated charge amount of the firstlight receiving element at the supplying timing of the charge readingout pulse voltage for the first time and the accumulated charge amountof the first light receiving element at the supplying timing of thecharge reading out pulse voltage for the second time are read out fromthe first light receiving element. It is to be noted that, within anexposure period, since transfer of charge of the vertical register 26stops, the charge amounts read out twice for each set are added by thevertical register 26. From the second light receiving element, theaccumulated charge amount of the second light receiving element at thesupplying timing of the charge reading out pulse voltage which issupplied only once for each set of the charge sweeping out pulse voltagefor the first and second times is read out. The charge amount read outonce for each set is added by the vertical register 26.

In such a second method for electronically implementing a sensitivitymosaic arrangement as described above, since reading out of charge isrepeated by a plural number of times within an exposure period,information of the subject over an overall region of the exposure periodcan be measured.

It is to be noted that, in connection with the first and second methodsfor electronically implementing a sensitivity mosaic arrangementdescribed above, reading out control of the CCD image sensor 4 isusually applied to the vertical register driving electrodes 25 wired foreach horizontal line. For example, in order to implement a sensitivitymosaic arrangement wherein the sensitivity changes for each horizontalline as in the color and sensitivity mosaic pattern P1 shown in FIG. 5,the electrode structure may be utilized, and therefore, someimprovements which allow application of different reading out pulsevoltages to different lines should be made. Further, in a CCD imagesensor of the progressive scanning type having a 3-phase driven verticalregister, an arbitrary mosaic arrangement with two different sensitivitystages can be implemented electronically by devising the electrodestructure.

FIG. 25 shows a first electrode structure of a poly-silicon electrodefor vertical transfer by an electrode wiring line used to implement asensitivity mosaic arrangement having two stages of sensitivity. FIG. 26shows a cross sectional view of the CCD image sensor taken along linea-a′ of FIG. 25. Each of a first phase vertical register drivingelectrode 42 and a second phase vertical register driving electrode 43is connected to electrodes of adjacent pixels on the same horizontalline, and therefore, the electrodes on the same horizontal line aredriven in synchronism. Meanwhile, a third phase vertical registerdriving electrode 44 is connected to electrodes of adjacent pixels onthe same vertical line, and therefore, the electrodes on the samevertical line are driven in synchronism. Further, the second phasevertical register driving electrode 43 and the third phase verticalregister driving electrode 44 overly a reading out gate 41 adjacent thecorresponding photo-diode 23.

Accordingly, when a reading out pulse is applied to the second phasevertical register driving electrode 43 or the third phase verticalregister driving electrode 44, the barrier of the reading out gate 41can be temporarily removed to allow charge accumulated in thecorresponding photo-diode 23 to be transferred to the vertical register26. In the following description, the electrode structure shown in FIGS.25 and 26 is referred to as OR type electrode structure.

FIG. 27 shows a second electrode structure of a polysilicon electrodefor vertical transfer by electrode wiring lines used to implement asensitivity mosaic arrangement having two stages of sensitivity. Alsothe cross section of the CCD image sensor taken along line a-a′ of FIG.27 is similar to that of the cross sectional view shown in FIG. 26. Inparticular, also in the second electrode structure, similarly to thefirst electrode structure, each of the first phase vertical registerdriving electrode 42 and the second phase vertical register drivingelectrode 43 is connected to electrodes of adjacent pixels on the samehorizontal line, and therefore, the electrodes on the same horizontalline are driven in synchronism. Since the third phase vertical registerdriving electrode 44 is connected to electrodes of adjacent pixels onthe same vertical line similarly as in the first electrode structure,the electrodes on the same vertical line are driven in synchronism.

However, the second electrode structure is different from the firstelectrode structure in that the third phase vertical register drivingelectrode 44 is disposed along an edge portion of the correspondingphoto-diode 23 on the reading out gate 41 adjacent the photo-diode 23and a portion of the second phase vertical register driving electrode 43which is worked in an elongated shape so as to be adjacent the edgeportion of the photo-diode 23 overlies the reading out gate 41.

Accordingly, when a reading out pulse is applied to only one of thesecond phase vertical register driving electrode 43 and the third phasevertical register driving electrode 44, the barrier of the reading outgate 41 cannot be removed. In order to remove the barrier of the readingout gate 41 to allow charge accumulated in the photo-diode 23 to betransferred to the vertical register 26, it is necessary to apply areading out pulse to the second phase vertical register drivingelectrode 43 and the third phase vertical register driving electrode 44simultaneously. In the following description, the electrode structureshown in FIG. 27 is referred to as AND type electrode structure.

An arbitrary mosaic arrangement with two stages of sensitivity can beproduced by using the OR type electrode structure and the AND typeelectrode structure described above in combination in one CCD imagesensor. For example, in order to implement a sensitivity mosaicarrangement of the color and sensitivity mosaic pattern P1 shown in FIG.5, the OR type electrode structure and the AND type electrode structureshould be used in such a combination as shown in FIG. 28.

As can be seen apparently from comparison between FIGS. 5 and 28, theAND type electrode structure is adopted for pixels having the lowsensitivity S0 from between the two sensitivity stages S0 and S1 whilethe OR type electrode structure is adopted for pixels of the highsensitivity S1. If the reading out pulse voltage is applied to thesecond phase vertical register driving electrodes 43 of the CCD imagesensor 4 formed from such a combination of the OR and AND type electrodestructures as just described, then charge reading out is performed onlywith the OR type pixels, but if the reading out pulse voltage is appliedto the second phase vertical register driving electrode 43 and the thirdphase vertical register driving electrode 44 simultaneously, then chargereading out is performed with both of the OR and AND type pixels, thatis, all pixels.

It is to be noted that, if the supplying timings of the pulse voltage tothe second phase vertical register driving electrode 43 and the thirdphase vertical register driving electrode 44 are such that both of thesecond phase and the third phase are driven at the supplying timing ofthe charge reading out pulse voltage for the first time in (D) of FIG.23 (or FIG. 24) from among the control timings shown in FIG. 23 (or FIG.24) and the supplying timing of the charge reading out pulse voltage of(F) of FIG. 23 (or FIG. 24) whereas only the second phase is driven atthe supplying timing of the charge reading out pulse voltage for thesecond time of (D) of FIG. 23 (or FIG. 24), then the pixels of the ORtype electrode structure have the high sensitivity S1 while the pixelsof the AND type electrode structure have the low sensitivity S0.

By a similar method, the other sensitivity mosaic arrangements havingtwo stages of sensitivity can be produced. For example, in order toimplement the sensitivity mosaic pattern of the color and sensitivitymosaic pattern P2 shown in FIG. 6, the OR type and the AND type are usedin such a combination as shown in FIG. 29. In order to implement thesensitivity mosaic pattern of the color and sensitivity mosaic patternP3 shown in FIG. 7, the OR type and the AND type are used in such acombination as shown in FIG. 30. In order to implement the sensitivitymosaic pattern of the color and sensitivity mosaic pattern P4 shown inFIG. 8, the OR type and the AND type are used in such a combination asshown in FIG. 31. In order to implement the sensitivity mosaic patternof the color and sensitivity mosaic pattern P5 shown in FIG. 9, the ORtype and the AND type are used in such a combination as shown in FIG.32.

Now, a demosaic process of the image processing system including theimage processing section 7 as a principal component is described.However, prior to the description of the demosaic process, a definitionof position coordinates of a pixel which is used in the descriptionhereinafter given is described with reference to FIG. 33.

FIG. 33 shows a coordinate system (x, y) indicating a position of apixel on an image. In particular, the left lower end of the image isrepresented by (0, 0) and the right upper end of the image isrepresented by (x_(max), y_(max)). Pixels represented by in FIG. 33 havea horizontal dimension and a vertical dimension of a length l and arearranged on a grating. Accordingly, for example, the coordinates of thecenter of the pixel at the left lower end are (0.5, 0.5), and thecoordinates of the center of the pixel at the right upper end are(x_(max)−0.5, y_(max)−0.5). Further, image data whose phase is displacedvertically and horizontally by a half pixel from the pixels representedby □ (pixel data at a position represented by ● in FIG. 33) is sometimesused, and, for example, the coordinates of image data whose phase isdisplaced vertically and horizontally by a half pixel from the pixel atthe left lower end are (1, 1).

FIG. 34 illustrates an outline of a first demosaic process of the imageprocessing system including the image processing section 7 as aprincipal component.

The first demosaic process includes, as seen in FIG. 34, a sensitivityuniformization process for uniformizing the sensitivities of pixels of acolor and sensitivity mosaic image obtained by processing of the imagepickup system without changing the colors of the pixels to produce acolor mosaic image, and a color correction process for restoring RGBcomponents of the pixels of a color and sensitivity mosaic image M.

An outline of the first sensitivity uniformization process in the firstdemosaic process is described with reference to FIGS. 35 to 37. FIGS. 35to 37 illustrate a pixel arrangement of a predetermined one line of animage to be processed. X0 represents that the color component is X (forexample, R (red)) and the sensitivity is S0 from between the two stagesof S0 and S1; X1 represents that the color component is X and thesensitivity is S1 from between the two stages of S0 and S1; Y0represents that the color component is Y (for example, G (green)) andthe sensitivity is S0 from between the two stages of S0 and S1; and Y1represents that the color component is Y and the sensitivity is S1 frombetween the two stages of S0 and S1. Each pixel of the sensitivity S0measures the intensity of incoming light attenuated at a predeterminedratio while each pixel of the sensitivity S1 measures the intensity ofincoming light without any attenuation. Further, in FIGS. 35 to 37, theaxis of abscissa indicates the position of a pixel on a line, and thelength of a vertical bar indicates the pixel value of a correspondingpixel.

The first sensitivity uniformization process in the first demosaicprocess can be divided into processes of two different stages. FIG. 35shows pixel values of pixels in a predetermined one line of a color andsensitivity mosaic image before the first sensitivity uniformizationprocess is performed. It is to be noted that a curve X indicates anintensity distribution of the color X of the incoming light, and anothercurve Y indicates an intensity distribution of the color Y.

A threshold value θ_(H) indicates a saturation level of the CCD imagesensor 4, and when the intensity of the incoming light exceeds thethreshold value θ_(H), the intensity cannot be measured accurately andthe measurement value then is equal to the threshold value θ_(H).Another threshold value θ_(L) indicates a noise level of the CCD imagesensor 4, and also when the intensity of the incoming light is lowerthan the threshold value θ_(L), the intensity cannot be measuredaccurately and the measurement value then is equal to the thresholdvalue θ_(L).

A validity discrimination result is information representative ofwhether or not each pixel has successfully measured the intensity of theincoming light, that is, information representative of whether the pixelvalue of each pixel measured is valid (V) or invalid (I).

Through the first stage process of the first sensitivity uniformizationprocess, the pixel values of the pixels of the sensitivity S0 are scaledusing the relative ratio of the sensitivity S0 to the sensitivity S1.The pixel values of the pixels of the sensitivity S1 are not scaled.FIG. 36 shows a result of application of the first stage process of thefirst sensitivity uniformization process. In the state after the firststage process is performed, as seen in FIG. 36, the pixels whosevalidity discrimination result is valid have an original light intensityrestored by the scaling, but the pixels whose validity discriminationresult is invalid do not have an original restored light intensity.

Therefore, in the second stage process of the first sensitivityuniformization process, the pixel value of each of those pixels whichare invalid is interpolated using the pixel values of those valid pixelsof the same color which neighbor with the pixel. FIG. 37 illustrates aresult of application of the second stage process of the firstsensitivity uniformization process. For example, the pixel of the colorY which is at the center of FIG. 37 and is invalid is interpolated inaccordance with an interpolation curve Y′ produced using the pixelvalues of those pixels of the color Y which neighbor with the pixel andare valid.

Subsequently, an outline of the second sensitivity uniformizationprocess of the first demosaic process is described with reference toFIGS. 35, 38 and 39. Also the second sensitivity uniformization processcan be divided into two stages of processes. The pixel values of pixelsin a predetermined one line of a color and sensitivity mosaic imagebefore the second sensitivity uniformization process is performed aresimilar to those in FIG. 35.

By the first stage process of the second sensitivity uniformizationprocess, pixel values with regard to the sensitivity S0 and pixel valueswith regard to the sensitivity S1 are estimated without changing thecolor of each pixel. For example, for a pixel of the sensitivity S0 ofthe color X, the pixel value with regard to the sensitivity S0 is usedat it is, and an estimated value with regard to the sensitivity S1 isinterpolated using the pixel values of those pixels of the sensitivityS1 and the color X which are present in the neighborhood of the pixel.FIG. 38 shows a result of application of the first stage process of thesecond sensitivity uniformization process. As shown in FIG. 38, afterthe first stage process is performed, each pixel has a pixel value ofsensitivity S0 or a pixel value of the sensitivity S1 of the originalcolor.

By the second stage process of the second sensitivity uniformizationprocess, for each pixel, the pixel values of the sensitivity S0 and thepixel values of the sensitivity S1 are synthesized to uniform thesensitivity. FIG. 39 shows a result of application of the second stageprocess of the second sensitivity uniformization process.

FIG. 40 shows an outline of the second demosaic process of the imageprocessing system which includes the image processing section 7 as aprincipal component.

The second demosaic process includes, as shown in FIG. 40, a sensitivityuniformization process wherein the colors of pixels of a color andsensitivity mosaic image obtained by the process of the image pickupsystem are changed to colors optimum for sensitivity uniformization andthe sensitivities are uniformized to produce a color mosaic image, and acolor correction process for restoring RGB components of pixels of thecolor and sensitivity mosaic image M.

An outline of the first sensitivity uniformization process of the seconddemosaic process is described with reference to FIGS. 35, 41 and 42.

Also the first sensitivity uniformization process of the second demosaicprocess can be divided into two stages of processes. It is assumed thatthe pixel values of pixels in a predetermined one line of a color andsensitivity mosaic image before the first sensitivity uniformizationprocess is performed are similar to those in FIG. 35.

Through the first stage process of the first sensitivity uniformizationprocess of the second demosaic process, the pixel values of the pixelsof the sensitivity S0 are scaled using the relative ratio of thesensitivity S0 to the sensitivity S1. The pixel values of the pixels ofthe sensitivity S1 are not scaled. FIG. 41 shows a result of applicationof the first stage process of the first sensitivity uniformizationprocess. In the state after the first stage process is performed, asseen in FIG. 41, the pixels whose validity discrimination result isvalid (V) have an original light intensity restored by the scaling, butthe pixels whose validity discrimination result is invalid (I) do nothave an original restored light intensity.

Therefore, in the second stage process of the first sensitivityuniformization process of the second demosaic process, the pixel valueof each of those pixels which are invalid is interpolated using thepixel values of those valid pixels, regardless colors thereof, whichneighbor with the pixel. FIG. 42 illustrates a result of application ofthe second stage process of the first sensitivity uniformizationprocess. For example, the pixel value of the pixel of the color Y whichis at the center of FIG. 41 and is invalid is interpolated in accordancewith an interpolation curve X′ produced using the pixel values of pixelsof the color X which neighbor with the pixel and are valid.

Now, an outline of the second sensitivity uniformization process of thesecond demosaic process is described with reference to FIGS. 35, 43 and44. Also the second sensitivity uniformization process of the seconddemosaic process can be divided into two stages of processes. It isassumed that the pixel values of pixels on a predetermined one line of acolor and sensitivity mosaic image before the second sensitivityuniformization process is performed are similar to those in FIG. 35.

In the first stage process of the second sensitivity uniformizationprocess of the second demosaic process, for each pixel, the pixel valuesof neighboring pixels which are positioned comparatively near to thepixel irrespective of the color are used to estimate the pixel valuewith regard to the sensitivity S0 and the pixel value with regard to thesensitivity S1. For example, as an estimated value of a pixel of thecolor X, where a pixel neighboring the pixel has the color Y, anestimated value with regard to the sensitivity S1 of the color Y and thepixel value with regard to the sensitivity S1 are interpolated. FIG. 43illustrates a result of application of the first stage process of thesecond sensitivity uniformization process. As shown in FIG. 43, afterthe first stage process is performed, each pixel has the pixel valuewith regard to the sensitivity S0 and the pixel value with regard to thesensitivity S1 of the original color because the color thereof has beenchanged to the color of the neighboring pixel irrespective of theoriginal color.

In the second stage process of the second sensitivity uniformizationprocess of the second demosaic process, for each pixel, the pixel valuewith regard to the sensitivity S0 and the pixel value with regard to thesensitivity S1 are synthesized to uniform the sensitivity. FIG. 44 showsa result of application of the second stage process of the secondsensitivity uniformization process.

Now, a first example of a configuration of the image processing section7 which principally executes the first demosaic process is describedwith reference to FIG. 45. It is assumed that, in the followingdescription, unless otherwise specified, the color and sensitivitymosaic image has the color and sensitivity mosaic pattern P2 of FIG. 6,or in other words, in the color and sensitivity mosaic image, the colorof each pixel is one of the three primary colors of R, G and B and thesensitivity is one of S0 and S1. However, the configuration and theoperation described below can be applied to another color andsensitivity mosaic image which includes three colors other than R, G andB or a further color and sensitivity mosaic image which includes fourcolors.

In the first example of a configuration of the image processing section7, a color and sensitivity mosaic image from the image pickup system issupplied to a sensitivity uniformization section 51. Color mosaicpattern information representative of a color mosaic arrangement of thecolor and sensitivity mosaic image is supplied to the sensitivityuniformization section 51 and a color interpolation section 52.Sensitivity mosaic pattern information representative of a sensitivitymosaic arrangement of the color and sensitivity mosaic image is suppliedto the sensitivity uniformization section 51.

The sensitivity uniformization section 51 performs a sensitivityuniformization process for the color and sensitivity mosaic image basedon the color mosaic pattern information and the sensitivity mosaicpattern information to produce a color mosaic image M wherein thesensitivities of the pixels are uniformized while the colors of thepixels are not changed, and outputs the color mosaic image M to thecolor interpolation section 52.

The color interpolation section 52 performs a color interpolationprocess, in which the color mosaic pattern information is used, for thecolor mosaic image M from the sensitivity uniformization section 51 toproduce output images R, G and B.

It is to be noted that the color mosaic pattern information isinformation representative of the types of the colors (in the presentcase, the colors of R, G and B) of the pixels of the color andsensitivity mosaic image, and information of a color component of eachof the pixels can be acquired using the position of the pixel as anindex.

The sensitivity mosaic pattern information is information representativeof the types of the sensitivities (in the present case, S0 and S1) ofthe pixels of the color and sensitivity mosaic image, and information ofthe sensitivity of each of the pixels can be acquired using the positionof the pixel as an index.

FIG. 46 shows a first example of the configuration of the sensitivityuniformization section 51. The first example of a configuration is anexample of a configuration of the sensitivity uniformization section 51which executes the first sensitivity uniformization process describedwith reference to FIGS. 35 to 37.

In the first example of the configuration of the sensitivityuniformization section 51, a color and sensitivity mosaic image from theimage pickup system is supplied to a sensitivity compensation section 61and a validity discrimination section 63. Color mosaic patterninformation is supplied to a missing interpolation section 64.Sensitivity mosaic pattern information is supplied to the sensitivitycompensation section 61 and the validity discrimination section 63.

The sensitivity compensation section 61 performs sensitivitycompensation for the color and sensitivity mosaic image based a relativesensitivity value S obtained from a relative sensitivity value LUT 62and outputs a resulting color and sensitivity mosaic image to themissing interpolation section 64. The relative sensitivity value LUT 62is a lookup table which outputs a relative sensitivity value S using thesensitivity of a pixel as an index.

The validity discrimination section 63 compares the pixel value of eachof the pixels of the color and sensitivity mosaic image with thethreshold value θ_(H) of the saturation level and the threshold valueθ_(L) of the noise level to discriminate the validity of the pixel valueand supplies a result of the discrimination as discriminationinformation to the missing interpolation section 64. In thediscrimination information, information representative of “valid” or“invalid” regarding the pixel value of each pixel is described.

The missing interpolation section 64 performs a missing interpolationprocess for the sensitivity-compensated color and sensitivity mosaicimage based on the discrimination information from the validitydiscrimination section 63 to produce a color mosaic image M and outputsthe color mosaic image M to the color interpolation section 52 in thenext stage.

FIG. 47 shows an example of a configuration of the color interpolationsection 52. In the color interpolation section 52, the color mosaicimage M from the sensitivity uniformization section 51 is supplied to agradation conversion section 71. The color mosaic pattern information issupplied to color difference image production sections 72 and 73 and aluminance image production section 74.

The gradation conversion section 71 performs a gradation conversionprocess for the color mosaic image M and supplies a resulting modulatedcolor mosaic image Mg to the color difference image production sections72 and 73 and the luminance image production section 74. For thegradation conversion process, particularly conversion based on a powerfunction of the power y or the like is used.

The color difference image production section 72 uses the modulatedcolor mosaic image Mg to produce a color difference image C wherein allpixels have a color difference C (=R−G) component and supplies the colordifference image C to the luminance image production section 74 and acolor space conversion section 75. The color difference image productionsection 73 produces a color difference image D wherein all pixels have acolor difference D (=B−G) component and supplies the color differenceimage D to the luminance image production section 74 and the color spaceconversion section 75. The luminance image production section 74 usesthe modulated mosaic image Mg and the color difference images C and D toproduce a luminance image L and supplies the luminance image L to thecolor space conversion section 75.

The color space conversion section 75 performs a color space conversionprocess for the color difference images C and D and the luminance imageL and supplies resulting modulated images (images in each of which thepixels have an R, G or B component) to gradation reverse conversionsections 76 to 78.

The gradation reverse conversion section 76 raises the pixel values ofthe R components from the color space conversion section 75 to the(1/γ)th power to perform reverse conversion to the gradation conversionby the gradation conversion section 71 to obtain an output image R. Thegradation reverse conversion section 77 raises the pixel values of the Gcomponents from the color space conversion section 75 to the (1/γ)thpower to perform reverse conversion to the gradation conversion by thegradation conversion section 71 to obtain an output image G. Thegradation reverse conversion section 78 raises the pixel values of the Bcomponents from the color space conversion section 75 to the (1/γ)thpower to perform reverse conversion to the gradation conversion by thegradation conversion section 71 to obtain an output image B.

It is to be noted that, where the color mosaic image M supplied from thesensitivity uniformization section 51 has a Bayer arrangement, the colorinterpolation section 52 may execute a color interpolation process, forexample, using the related-art method disclosed in the official gazetteof Japanese Patent Laid-Open No. Sho 61-501424 and so forth.

FIG. 48 shows an example of a configuration of the color differenceimage production section 72. In the color difference image productionsection 72, the modulated color mosaic image Mg from the gradationconversion section 71 is supplied to smoothing sections 81 and 82. Alsothe color mosaic pattern information is supplied to the smoothingsections 81 and 82.

The smoothing section 81 uses, for each pixel, the pixel values ofneighboring pixels having an R component to interpolate the R componentof the pixel to produce a smoothed image R′ of the R component andsupplies the image R′ to a subtractor 83. The smoothing section 82 uses,for each pixel, the pixel values of neighboring pixels having a Gcomponent to interpolate the G component of the pixel to produce asmoothed image G′ of the G component and supplies the image G′ to thesubtractor 83.

The subtractor 83 subtracts the pixel values of the pixels of thesmoothed image G′ of the G component from the smoothing section 82 fromthe pixel values of the corresponding pixels of the smoothed image R′ ofthe R component from the smoothing section 81 to produce a colordifference image C and supplies the color difference image C to thecolor space conversion section 75.

It is to be noted that also the color difference image productionsection 73 has a similar configuration.

FIG. 49 shows an example of a configuration of the luminance imageproduction section 74. A luminance calculation section 91 which composesthe luminance image production section 74 calculates a luminancecandidate value of each pixel based on the modulated color mosaic imageMg from the gradation conversion section 71, the color difference imageC from the color difference image production section 72, the colordifference image D from the color difference image production section 73and the color mosaic pattern information and outputs a luminancecandidate value image Lc formed from luminance pixel values of thepixels to a noise removal section 92.

The noise removal section 92 synthesizes a smoothing component(hereinafter described) with each of the pixel values (luminancecandidate values) of the luminance candidate value image Lc to removenoise from the luminance candidate value image Lc and outputs aresulting luminance image L to the color space conversion section 75.

Subsequently, the first demosaic process by the first example of theconfiguration of the image processing section 7 shown in FIG. 45 isdescribed with reference to a flow chart of FIG. 50.

At step S1, the sensitivity uniformization section 51 performs asensitivity uniformization process for the color and sensitivity mosaicimage based on the color mosaic pattern information and the sensitivitymosaic pattern information and outputs a resulting color mosaic image Mto the color interpolation section 52.

Details of the first sensitivity uniformization process by the firstexample of the configuration of the sensitivity uniformization section51 shown in FIG. 46 are described with reference to a flow chart of FIG.51.

At step S11, the sensitivity compensation section 61 performs asensitivity compensation process for the color and sensitivity mosaicimage inputted thereto and supplies the sensitivity-compensated colorand sensitivity mosaic image to the missing interpolation section 64.

Details of the sensitivity compensation process are described withreference to a flow chart of FIG. 52. At step S21, the sensitivitycompensation section 61 discriminates whether or not all pixels of thecolor and sensitivity mosaic image have been used as a noticed pixel. Ifthe sensitivity compensation section 61 discriminates that all pixelshave not been used as a noticed pixel, then the processing advances tostep S22. At step S22, the sensitivity compensation section 61determines one by one pixel as a noticed pixel beginning with the leftlowermost pixel and ending with the right uppermost pixel of the colorand sensitivity mosaic image.

At step S23, the sensitivity compensation section 61 refers to thesensitivity mosaic pattern information to acquire the sensitivity (S0 orS1) of the noticed pixel and further refers to the relative sensitivityvalue LUT 62 to acquire the relative sensitivity value S correspondingto the pixel of the noticed pixel.

At step S24, the sensitivity compensation section 61 divides the pixelvalue of the noticed pixel of the color and sensitivity mosaic image bythe relative sensitivity value S to compensate for the sensitivity ofthe pixel value. The sensitivity-compensated pixel value is a pixelvalue of a sensitivity-compensated color and sensitivity mosaic image.

The processing returns to step S21 so that the processing at steps S21to S24 is repeated until it is discriminated at step S21 that all pixelshave been used as a noticed pixel. When it is discriminated at step S21that all pixels have been used as a noticed pixel, the processingreturns to step S12 of FIG. 51.

At step S12, the validity discrimination section 63 performs a validitydiscrimination process for the color and sensitivity mosaic image toproduce discrimination information representative of the validity of thepixel value of each pixel and supplies the discrimination information tothe missing interpolation section 64. It is to be noted that thevalidity discrimination process at step S12 may be executed in parallelto the sensitivity compensation process at step S61.

Details of the validity discrimination process are described withreference to a flow chart of FIG. 53. At step S31, the validitydiscrimination section 63 discriminates whether or not all pixels of thecolor and sensitivity mosaic image have been used as a noticed pixel. Ifit is discriminated that all pixels have not been used as a noticedpixel, then the processing advances to step S32. At step S32, thevalidity discrimination section 63 determines one by one pixel as anoticed pixel beginning with the left lowermost pixel and ending withthe right uppermost pixel of the color and sensitivity mosaic image.

At step S33, the validity discrimination section 63 discriminateswhether or not the pixel value of the noticed pixel of the color andsensitivity mosaic image is within the range between the threshold valueθ_(L) of the noise level and the threshold value θ_(H) of the saturationlevel. If the validity discrimination section 63 discriminates that thepixel value is within the range between the threshold values, then theprocessing advances to step S34.

At step S34, the validity discrimination section 63 sets thediscrimination information of the noticed pixel as valid. The processingreturns to step S31.

If it is discriminated at step S33 that the pixel value of the noticedpixel of the color and sensitivity mosaic image is not within the rangebetween the threshold values, then the processing advances to step S35.At step S35, the validity discrimination section 63 discriminateswhether or not the pixel value of the noticed pixel of the color andsensitivity mosaic image is equal to or higher than the threshold levelθ_(H) of the saturation level. If the validity discrimination section 63discriminates that the pixel value is higher than the threshold valueθ_(H) of the saturation level, then the processing advances to step S36.

At step S36, the validity discrimination section 63 refers to thesensitivity mosaic pattern information to discriminate whether or notthe noticed pixel has the sensitivity S0. If the validity discriminationsection 63 discriminates that the noticed pixel has the sensitivity S0,then the processing advances to step S34. If the validity discriminationsection 63 discriminates that the noticed pixel does not have thesensitivity S0, then the processing advances to step S37.

At step S37, the validity discrimination section 63 sets thediscrimination information of the noticed pixel as invalid. Theprocessing returns to step S31.

If it is discriminated at step S35 that the pixel value of the noticedpixel of the color and sensitivity mosaic image is not equal to orhigher than the threshold value θ_(H) of the saturation level, then theprocessing advances to step S38. At step S38, the validitydiscrimination section 63 refers to the sensitivity mosaic patterninformation to discriminate whether or not the noticed pixel has thesensitivity S1. If the validity discrimination section 63 discriminatesthat the noticed pixel has the sensitivity S1, then the processingadvances to step S34. However, if the validity discrimination section 63discriminates that the noticed pixel does not have the sensitivity S1,then the processing advances to step S37.

Thereafter, the processing at steps S31 to S38 is repeated until it isdiscriminated at step S31 that all pixels have been used as a noticedpixel. When it is discriminated at step S31 that all pixels have beenused as a noticed pixel, the processing returns to step S13 of FIG. 51.

At step S13, the missing interpolation section 64 performs a missinginterpolation process for the sensitivity-compensated color andsensitivity mosaic image based on the discrimination information fromthe validity discrimination section 63 and supplies a resulting colormosaic image M to the color interpolation section 52.

Details of the missing interpolation process are described withreference to a flow chart of FIG. 54. At step S41, the missinginterpolation section 64 discriminates whether or not all pixels of thesensitivity-compensated color and sensitivity mosaic image have beenused as a noticed pixel. If the missing interpolation section 64discriminates that all pixels have not been used as a noticed pixel,then the processing advances to step S42. At step S42, the missinginterpolation section 64 determines one by one pixel as a noticed pixelbeginning with the left lowermost pixel and ending with the rightuppermost pixel of the sensitivity-compensated color and sensitivitymosaic image.

At step S43, the missing interpolation section 64 discriminates whetheror not the discrimination information of the noticed pixel is invalid.If the missing interpolation section 64 discriminates that thediscrimination information is invalid, then the processing advances tostep S44.

At step S44, the missing interpolation section 64 refers to the colormosaic pattern information to discriminate the type of the color of thenoticed pixel (in the present case, one of the colors of R, G and B),detect, from among neighboring pixels with the noticed pixel (forexample, in the present case, 5×5 pixels centered at the noticed pixel),those pixels which have the same color and whose discriminationinformation is valid, and extracts the pixel values of the detectedpixels (hereinafter referred to as reference pixels).

At step S45, the missing interpolation section 64 acquires a number offilter coefficients set in advance corresponding to relative positionsof the reference pixels to the noticed pixel, the number being equal tothe number of the reference pixels. At step S46, the missinginterpolation section 64 multiplies the pixel values of the referencepixels by the corresponding filter coefficients and arithmeticallyoperates the sum total of the products. Further, the missinginterpolation section 64 divides the sum total of the products by thesum total of the used filter coefficients and determines the quotient asa pixel value of the noticed pixel of the color mosaic image M.

The processing returns to step S41 so that the processing at steps S41to 46 is repeated until it is discriminated at step S41 that all pixelshave been used as a noticed pixel. When it is discriminated at step S41that all pixels have been used as a noticed pixel, the processingreturns to step S2 of FIG. 50.

At step S2, the color interpolation section 52 performs a colorinterpolation process for the color mosaic image M obtained by thesensitivity uniformization process at step S1 described above based onthe color mosaic pattern information to produce output images R, G andB.

Details of the color interpolation process are described with referenceto a flow chart of FIG. 55. At step S51, the gradation conversionsection 71 performs a gradation modulation process for the color mosaicimage M (more particularly, raises the pixel values of the modulatedcolor mosaic image Mg to the γth power) to produce a modulated colormosaic image Mg and supplies the modulated color mosaic image Mg to thecolor difference image production sections 72 and 73 and the luminanceimage production section 74.

At step S52, the color difference image production section 72 uses themodulated color mosaic image Mg from the gradation conversion section 71to produce a color difference image C and supplies the color differenceimage C to the luminance image production section 74 and the color spaceconversion section 75. Meanwhile, the color difference image productionsection 73 uses the modulated color mosaic image Mg from the gradationconversion section 71 to produce a color difference image D and suppliesthe color difference image D to the luminance image production section74 and the color space conversion section 75.

The first process of the color difference image production section 72producing a color difference image C is described with reference to aflow chart of FIG. 56. At step S61, the smoothing sections 81 and 82discriminate whether or not all pixels of the modulated color mosaicimage Mg have been used as a noticed pixel. If the smoothing sections 81and 82 discriminate that all pixels have not been used as a noticedpixel, then the processing advances to step S62. At step S62, thesmoothing sections 81 and 82 determine one by one pixel as a noticedpixel beginning with the left lowermost pixel and ending with the rightuppermost pixel of the modulated color mosaic image Mg.

At step S63, the smoothing section 81 refers to the color mosaic patterninformation to detect, from among neighboring pixels with the noticedpixel (for example, 5×5 pixels centered at the noticed pixel), thosepixels which have an R component, and extracts the pixel values of thedetected pixels (hereinafter referred to as reference pixels).Meanwhile, also the smoothing section 82 similarly refers to the colormosaic pattern information to detect, from among neighboring pixels withthe noticed pixel, those pixels which have a G component, and extractsthe pixel values of the detected pixels.

At step S64, the smoothing section 81 acquires a number of filtercoefficients set in advance corresponding to relative positions of thereference pixels having an R component to the noticed pixel, the numberbeing equal to the number of the reference pixels. Meanwhile, also thesmoothing section 82 similarly acquires a number of filter coefficientsset in advance corresponding to relative positions of the referencepixels having a G component to the noticed pixel, the number being equalto the number of the reference pixels.

At step S65, the smoothing section 81 multiplies the pixel values of thereference pixels having an R component by the corresponding filtercoefficients and arithmetically operates the sum total of the products.Further, the smoothing section 81 divides the sum total of the productsby the sum total of the used filter coefficients and determines thequotient as a pixel value corresponding to the noticed pixel of an imageR′ which includes only smoothed R components. Meanwhile, also thesmoothing section 82 similarly multiplies the pixel values of thereference pixels having a G component by the corresponding filtercoefficients and arithmetically operates the sum total of the products.Further, the smoothing section 82 divides the sum total of the productsby the sum total of the used filter coefficients and determines thequotient as a pixel value corresponding to the noticed pixel of an imageG′ which includes only smoothed G components.

At step S66, the subtractor 83 subtracts the pixel value correspondingto the noticed pixel of the image R′ which includes only smoothed Rcomponents from the smoothing section 81 from the pixel valuecorresponding to the noticed pixel of the image G′ which includes onlysmoothed G components from the smoothing section 82 and determines thedifference as a pixel value corresponding to the noticed pixel of acolor difference image C.

The processing returns to step S61 so that the processing at steps S61to S66 is repeated until it is discriminated at step S61 that all pixelshave been used as a noticed pixel. When it is discriminated at step S61that all pixels have been used as a noticed pixel, the processingreturns to step S53 of FIG. 55.

It is to be noted that, since the processing of the color differenceimage production section 73 when it produces a color difference image Dis similar to the first process of the color difference image productionsection 72 when it produces the color difference image C describedabove, description of the processing is omitted.

At step S53, the luminance image production section 74 produces aluminance image L using the modulated mosaic image Mg and the colordifference signals C and D and supplies the luminance image L to thecolor space conversion section 75.

Details of the luminance image production process of the luminance imageproduction section 74 are described with reference to a flow chart ofFIG. 57. At step S71, the luminance calculation section 91 discriminateswhether or not all pixels of the modulated color mosaic image Mg havebeen used as a noticed pixel. If the luminance calculation section 91discriminates that all pixels have not been used as a noticed pixel,then the processing advances to step S72. At step S72, the luminancecalculation section 91 determines one by one pixel as a noticed pixelbeginning with the left lowermost pixel and ending with the rightuppermost pixel of the modulated color mosaic image Mg.

At step S73, the luminance calculation section 91 refers to the colormosaic pattern information to discriminate the type of the color of thenoticed pixel (in the present case, one of the colors of R, G and B).

If it is discriminated at step S73 that the type of the color of thenoticed pixel is R, then the processing advances to step S74. At stepS74, the luminance calculation section 91 applies the modulated colormosaic image Mg and the pixel values of the color difference signals Cand D corresponding to the noticed pixel to the following expression (1)to calculate the pixel value of a luminance candidate image Lccorresponding to the noticed pixel:Lc=3Mg−2C+D   (1)

If it is discriminated at step S73 that the type of the color of thenoticed pixel is G, then the processing advances to step S75. At stepS75, the luminance calculation section 91 applies the modulated colormosaic image Mg and the pixel values of the color difference signals Cand D corresponding to the noticed pixel to the following expression (2)to calculate the pixel value of the luminance candidate image Lccorresponding to the noticed pixel:Lc=3Mg+C+D   (2)

If it is discriminated at step S73 that the type of the color of thenoticed pixel is B, then the processing advances to step S76. At stepS76, the luminance calculation section 91 applies the modulated colormosaic image Mg and the pixel values Mg of the color difference signalsC and D corresponding to the noticed pixel to the following expression(3) to calculate the pixel value of the luminance candidate image Lccorresponding to the noticed pixel:Lc=3Mg+C−2D   (3)

It is to be noted that, in the expressions (1) to (3), Lc, Mg, C and Drepresent the pixel values of the luminance candidate image Lc,modulated color mosaic image Mg, color difference signal C and colordifference image D corresponding to the noticed pixel, respectively.

The processing returns to step S71 so that the processing at steps S71to S76 is repeated until it is discriminated at step S71 that all pixelshave been used as a noticed pixel. When it is discriminated at step S71that all pixels have been used as a noticed pixel, the processingadvances to step S77.

The luminance candidate image Lc produced by the processing at steps S71to S76 described above is supplied to the noise removal section 92.

At step S77, the noise removal section 92 discriminates whether or notall pixels of the modulated color mosaic image Mg have been used as anoticed pixel. If the noise removal section 92 discriminates that allpixels have not been used as a noticed pixel, then the processingadvances to step S78. At step S78, the noise removal section 92determines one by one pixel as a noticed pixel beginning with the leftlowermost pixel and ending with the right uppermost pixel of themodulated color mosaic image Mg.

At step S79, the noise removal section 92 applies the pixel values(luminance candidate values) of the pixels positioned upwardly,downwardly, leftwardly and rightwardly of the noticed pixel to thefollowing expression (4) to calculate a gradient ∇ corresponding to thenoticed pixel. It is to be noted that the gradient ∇ is a vector whosefactors are linear differential coefficients in the horizontal directionand the vertical direction of the image. Further, the pixel values(luminance candidate values) of the pixels positioned upwardly,downwardly, leftwardly and rightwardly of the noticed pixel arerepresented by Lc(U), Lc(D), Lc(L) and Lc(R), respectively.gradient ∇=(Lc(R)−Lc(L),Lc(U)−Lc(D))   (4)

At step S80, the noise removal section 92 applies the pixel values(luminance candidate values) of the pixels positioned leftwardly,rightwardly, upwardly and downwardly of the noticed pixel to thefollowing expressions (5) and (6) to calculate a smoothed component Hhin the horizontal direction and a smoothed component Hv in the verticaldirection corresponding to the noticed pixel:Hh=(Lc(L)+Lc(R))/2   (5)Hν=(Lc(U)+Lc(D))/2   (6)

At step S81, the noise removal section 92 calculates a smoothingcontribution wh in the horizontal direction and a smoothing contributionwv in the vertical direction corresponding to the absolute value

∥∇∥ of the gradient ∇ corresponding to the noticed pixel calculated atstep S79.

More particularly, where the absolute value of the gradient ∇ is higherthan 0, the absolute value of the inner product of the normalizedgradient ∇/∥∇∥ and the vector (1, 0) is subtracted from 1 as given bythe following expression (7) to obtain the smoothing contribution wh inthe horizontal direction. Further, as given by the following expression(8), the absolute value of the inner product of the normalized gradient∇/∥∇∥ and the vector (0, 1) is subtracted from 1 to obtain the smoothingcontribution wv in the vertical direction:wh=1−|∇/∥∇∥,(1,0)|  (7)wv=1−|∇/∇∥∇∥,(0,1)|  (8)

Where the absolute value of the gradient ∇ is 0, the smoothingcontribution wh in the horizontal direction and the smoothingcontribution wv in the vertical direction are both set to 0.5.

At step S82, the noise removal section 92 uses the following expression(9) to calculate the pixel value (luminance value) of the luminanceimage L corresponding to the noticed pixel:L=Lc+(wh·Hh+wv·Hv)/(wh+wv)   (9)

It is to be noted that Lc and L in the expression (9) represent thepixel values of the luminance candidate image Lc and the luminance imageL corresponding to the noticed pixel.

The processing returns to step S77 so that the processing at steps S77to S82 is repeated until it is discriminated at step S77 that all pixelshave been used as a noticed pixel. When it is discriminated at step S77that all pixels have been used as a noticed pixel, the processingreturns to step S54 of FIG. 55.

At step S54, the color space conversion section 75 performs a colorspace conversion process for the color difference images C and D and theluminance image L to produce modulated images in each of which eachpixel has an R, G or B component and supplies the modulated images tothe gradation reverse conversion sections 76 to 78, respectively.

Details of the color space conversion process are described withreference to a flow chart of FIG. 58. At step S91, the color spaceconversion section 75 discriminates whether or not all pixels of theluminance image L (which may alternatively be the color difference imageC or the color difference image D) have been used as a noticed pixel. Ifthe color space conversion section 75 discriminates that all pixels havenot been used as a noticed pixel, then the processing advances to stepS92. At step S92, the color space conversion section 75 determines oneby one pixel as a noticed pixel beginning with the left lowermost pixeland ending with the right uppermost pixel of the color and sensitivitymosaic image.

At step S93, the color space conversion section 75 applies the pixelvalues of the luminance image L, color difference image C and colordifference image D corresponding to the noticed pixel to the followingexpressions (10), (11) and (12) to calculate the value Rg of the Rcomponent, the value Gg of the G component and the value Bg of the Bcomponent of the modulated images corresponding to the noticed pixel:Rg=(L+2C−D)/3   (10)Gg=(L−C−D)/3   (11)Bg=(L−C+2D)/3   (12)

It is to be noted that, in the expressions (10) to (12), L, C and D arethe pixel values of the luminance image L, color difference signal C andcolor difference image D corresponding to the noticed pixel,respectively.

The processing returns to step S91 so that the processing at steps S91to S93 is repeated until it is discriminated at step S91 that all pixelshave been used as a noticed pixel. When it is discriminated at step S91that all pixels have been used as a noticed pixel, the processingreturns to step S55 of FIG. 55.

At step S55, the gradation reverse conversion section 76 performs agradation reverse conversion process corresponding to the gradationconversion process at step S51 (more particularly, to raise pixel valuesto the 1/γth power) for the R component of each pixel of the modulatedimage supplied from the color space conversion section 75 to produce anoutput image R. Similarly, the gradation reverse conversion section 77performs a gradation reverse conversion process corresponding to thegradation conversion process at step S51 for the G component of eachpixel of the modulated image supplied from the color space conversionsection 75 to produce an output image G. The gradation reverseconversion section 78 performs a gradation reverse conversion processcorresponding to the gradation conversion process at step S51 for the Bcomponent of each pixel of the modulated image supplied from the colorspace conversion section 75 to produce an output image B. Through such acolor interpolation process as described above, the output images R, Gand BG are produced.

Description of the first demosaic process by the first example of theconfiguration of the sensitivity uniformization section 51 shown in FIG.45 is ended thereby.

Now, a second example of the configuration of the sensitivityuniformization section 51 which can be used in place of the secondexample of the configuration of the sensitivity uniformization section51 shown in FIG. 46 is described with reference to FIG. 59.

The second example of the configuration is an example of theconfiguration wherein the second sensitivity uniformization process inthe first demosaic process described with reference to FIGS. 35, 38 and39 is executed by the sensitivity uniformization section 51.

It is assumed that, in the color and sensitivity mosaic image describedbelow, the color of each pixel is one of the three primary colors of R,G and B and the sensitivity is one of four stages S0, S1, S2 and S3 asin the color and sensitivity mosaic pattern P10 of FIG. 14 or the colorand sensitivity mosaic pattern P1 of FIG. 15. However, the configurationand the operation described below can be applied also to another colorand sensitivity mosaic image which includes three colors other than R, Gand B or a further color and sensitivity mosaic image which includesfour colors. Further, they can be applied also to a color andsensitivity mosaic pattern wherein the sensitivity has two stages orthree stages.

In the second example of the configuration of the sensitivityuniformization section 51, a color and sensitivity mosaic image from theimage pickup system, color mosaic pattern information and sensitivitymosaic pattern information are supplied to interpolation sections 101-1to 101-4.

The interpolation section 101-1 performs an interpolation process of thesensitivity S0 without changing the color of each pixel of the color andsensitivity mosaic image and outputs an interpolation valuecorresponding to the resulting sensitivity S0 to an adder 102. Theinterpolation section 101-2 performs an interpolation process of thesensitivity S1 without changing the color of each pixel of the color andsensitivity mosaic image and outputs an interpolation valuecorresponding to the resulting sensitivity S1 to the adder 102. Theinterpolation section 101-3 performs an interpolation process of thesensitivity S2 without changing the color of each pixel of the color andsensitivity mosaic image and outputs an interpolation valuecorresponding to the resulting sensitivity S2 to the adder 102. Theinterpolation section 101-4 performs an interpolation process of thesensitivity S3 without changing the color of each pixel of the color andsensitivity mosaic image and outputs an interpolation valuecorresponding to the resulting sensitivity S3 to the adder 102.

The adder 102 adds, for each pixel, the sensitivities S0 to S3 inputtedthereto from the interpolation sections 101-1 to 101-4 and supplies thesum as a pixel value of a color mosaic candidate image to a syntheticsensitivity compensation section 103.

The synthetic sensitivity compensation section 103 collates the pixelvalue of the color mosaic candidate image supplied thereto from theadder 102 with a synthetic sensitivity compensation LUT 104 to produce acolor mosaic image M wherein the resulting value is used as a pixelvalue and supplies the color mosaic image M to the color interpolationsection 52. The synthetic sensitivity compensation LUT 104 is configuredso as to acquire a pixel value of the color mosaic image M using a pixelvalue of the color mosaic candidate image as an index.

The second sensitivity uniformization process in the first demosaicprocess by the second example of the configuration of the sensitivityuniformization section 51 shown in FIG. 59 is described with referenceto a flow chart of FIG. 60.

At step S101, the interpolation sections 101-1 to 101-4 discriminatewhether or not all pixels of the color and sensitivity mosaic image havebeen used as a noticed pixel. If the interpolation sections 101-1 to101-4 discriminate that all pixels have not been used as a noticedpixel, then the processing advances to step S102. At step S102, theinterpolation sections 101-1 to 101-4 determine one by one pixel as anoticed pixel beginning with the left lowermost pixel and ending withthe right uppermost pixel of the color and sensitivity mosaic image.

At step S103, the interpolation sections 101-1 to 101-4 perform aninterpolation process without changing the color of each pixel of thecolor and sensitivity mosaic image to produce interpolation valuescorresponding to the sensitivities S0, S1, S2 and the sensitivity S3,respectively, and output the interpolation values to the adder 102.

The interpolation process for the sensitivity S0 by the interpolationsection 101-1 is described with reference to a flow chart of FIG. 61. Atstep 111, the interpolation section 101-1 detects those of pixelspositioned in the neighborhood of the noticed pixel of the color andsensitivity mosaic image (for example, 5×5 pixels centered at thenoticed pixel) which have a color same as that of the noticed pixel andhave the sensitivity S0, and extracts the pixel values of the detectedpixels (hereinafter referred to as reference pixels). At step S112, theinterpolation section 101-1 acquires a number of filter coefficients setin advance corresponding to relative positions of the detected referencepixels to the noticed pixel, the number being equal to the number of thereference pixels. At step S113, the interpolation section 101-1multiplies the pixel values of the reference pixels and thecorresponding filter coefficients and arithmetically operates the sumtotal of the products. Further, the interpolation section 101-1 dividesthe sum total of the products by the sum total of the used filtercoefficients and determines the quotient as an interpolation valuecorresponding to the sensitivity S0 of the noticed pixel. The processingreturns to step S60 of FIG. 60.

It is to be noted that, since the interpolation processes for thesensitivities S1 to S3 by the interpolation sections 101-2 and 101-3 aresimilar to the interpolation process for the sensitivity S0 by theinterpolation section 101-1 described above, description of theinterpolation processes is omitted.

At step S104, the adder 102 adds the interpolation values for thesensitivities S0 to S3 corresponding to the noticed pixel inputted fromthe interpolation sections 101-1 to 101-4 and supplies the sum as apixel value of a color mosaic candidate image corresponding to thenoticed pixel to the synthetic sensitivity compensation section 103.

At step S105, the synthetic sensitivity compensation section 103collates the pixel value of the color mosaic candidate image suppliedthereto from the adder 102 with the synthetic sensitivity compensationLUT 104 and determines a detected value as a pixel value of a colormosaic image M corresponding to the noticed pixel.

The processing returns to step S101 so that the processing at steps S101to S105 is repeated until it is discriminated at step S101 that allpixels have been used as a noticed pixel. When it is discriminated atstep S101 that all pixels have been used as a noticed pixel, the secondsensitivity uniformization process of the first demosaic process isended.

It is to be noted that, after the second sensitivity uniformizationprocess, the color interpolation process described hereinabove withreference to the flow chart of FIG. 55 is executed.

Now, a second process for producing a color difference image C which canbe executed by the color difference image production section 72 in placeof the first process (FIG. 56) for producing a color difference image Cdescribed hereinabove is described with reference to a flow chart ofFIG. 62.

At step S121, the smoothing sections 81 and 82 discriminate whether ornot all pixels of the modulated color mosaic image Mg have been used asa noticed pixel. If the smoothing sections 81 and 82 discriminate thatall pixels have not been used as a noticed pixel, then the processingadvances to step S122. At step S122, the smoothing sections 81 and 82determine one by one pixel as a noticed pixel beginning with the leftlowermost pixel and ending with the right uppermost pixel of themodulated color mosaic image Mg.

At step S123, the smoothing section 81 arithmetically operates an imagegradient vector g corresponding to the noticed pixel.

Details of the image gradient vector arithmetic operation process aredescribed with reference to a flow chart of FIG. 63. In the imagegradient vector arithmetic operation process, only those of all pixelsof the color mosaic image Mg which have a single type of a color areused to arithmetically operate the image gradient vector g.

It is to be noted that, although a predetermined single type of a colormay be selected arbitrarily, for example, where the color mosaic patternof the color mosaic image Mg has a Bayer arrangement, since the numberof pixels having a G component is equal to twice that of pixels havingan R component or pixels having a B component, the single type of acolor is reasonably set to G. Accordingly, the following descriptionproceeds assuming that the color mosaic pattern of the color mosaicimage Mg has a Bayer arrangement and that G is selected as thepredetermined single type of a color.

At step S141, the smoothing section 81 discriminates whether or not thecolor of the noticed pixel is G. If the smoothing section 81discriminates that the color of the noticed pixel is G, then theprocessing advances to step S142. In this instance, the colors of thefour pixels positioned upwardly, downwardly, leftwardly and rightwardlyof the noticed pixel are not G, and the colors of the four pixelspositioned in the oblique directions from the noticed pixel are G.

At step S142, the smoothing section 81 interpolates the values G(U),G(D), G(L) and G(R) of G components corresponding to the four pixelspositioned upwardly, downwardly, leftwardly and rightwardly of thenoticed pixel, respectively, by applying the pixel value G(LU) of thepixel neighboring leftwardly upwards of the noticed pixel and having a Gcomponent, the pixel value G(LD) of the pixel neighboring leftwardlydownwards of the noticed pixel and having a G component, the pixel valueG(RU) of the pixel neighboring rightwardly upwards of the noticed pixeland having a G component and the pixel value G(RD) of the pixelneighboring rightwardly downwards of the noticed pixel and having a Gcomponent to the following expressions (13) to (16):G(U)=(G(LU)+G(RU))/2   (13)G(D)=(G(LD)+G(RD))/2   (14)G(L)=(G(LU)+G(LD))/2   (15)G(R)=(G(RU)+G(RD))/2   (16)

At step S143, the smoothing section 81 applies the values G(U), G(D),G(L) and G(R) of the G components corresponding to the four pixelspositioned upwardly, downwardly, leftwardly and rightwardly of thenoticed pixel to the following expressions (17) to (19) to calculate avector g′ and normalize the vector g′ in accordance with the followingexpression (20) to calculate a gradient vector g:gh=G(R)−G(L)   (17)gv=G(U)−G(D)   (18)g′=(gh,gv)   (19)g=G′/∥g′∥  (20)

It is to be noted that, if it is discriminated at step S141 that thecolor of the noticed pixel is not G, then the processing advances tostep S144. In this instance, the colors of the four pixels positionedupwardly, downwardly, leftwardly and rightwardly of the noticed pixelare G.

At step S144, the smoothing section 81 acquires the pixel values of thefour pixels positioned upwardly, downwardly, leftwardly and rightwardlyof the noticed pixel and substitutes them into the values G(U), G(D),G(L) and G(R), respectively.

The image gradient vector g corresponding to the noticed pixel isarithmetically operated in such a manner as described above. It is to benoted that, also where the color mosaic pattern of the color mosaicimage Mg does not have a Bayer arrangement, a similar process can beapplied to arithmetically operate the image gradient vector g.

The processing returns to step S124 of FIG. 62.

At step S124, the smoothing section 81 refers to the color mosaicpattern information to detect those of pixels neighboring with thenoticed pixel (for example, 5×5 pixels centered at the noticed pixel)which have an R component, and extracts the pixel values of the detectedpixels (hereinafter referred to as reference pixels). Meanwhile, alsothe smoothing section 82 similarly refers to the color mosaic patterninformation to detect those of pixels neighboring with the noticed pixelwhich have a G component, and extracts the pixel values of the detectedpixels.

At step S125, the smoothing section 81 calculates the position vectors nfrom the noticed pixel to the reference pixels which have an R componentand normalizes them. Meanwhile, also the smoothing section 82 similarlycalculates the position vectors n from the noticed pixel to thereference pixels which have a G component and normalizes them.

At step S126, as shown in the following expression (21), the smoothingsection 81 divides, for each of the reference pixels having an Rcomponent, the absolute value of an inner product of the gradient vectorg of the noticed pixel and the position vector n from 1 andarithmetically operates the difference to the ρth power to calculate asignificance ω of the reference pixel. Meanwhile, also the smoothingsection 82 similarly calculates a significance ω for each of thereference pixels having a G component. Here, ρ is a constant foradjusting the sharpness of direction selection and is set in advance.ω=(1−|(n,g)|)^(ρ)  (21)

At step S127, the smoothing section 81 acquires a number of filtercoefficients set in advance corresponding to relative positions of thereference pixels having an R component to the noticed pixel, the numberbeing equal to the number of the reference pixels. Meanwhile, also thesmoothing section 82 similarly acquires a number of filter coefficientsset in advance corresponding to relative positions of the referencepixels having a G component to the noticed pixel, the number being equalto the number of the reference pixels.

At step S128, the smoothing section 81 multiplies the pixel values ofthe reference pixels having an R component by the corresponding filtercoefficients and significances ω and arithmetically operates the sumtotal of the products. Further, the smoothing section 81 multiplies thefilter coefficients and the significances ω corresponding to thereference pixels and arithmetically operates the sum total of theproducts. Meanwhile, also the smoothing section 82 similarly multipliesthe pixel values of the reference pixels having a G component by thecorresponding filter coefficients and significances ω and arithmeticallyoperates the sum total of the products. Further, the smoothing section82 multiplies the filter coefficients and the significances ωcorresponding to the reference pixels and arithmetically operates thesum total of the products.

At step S129, the smoothing section 81 divides the sum total of theproducts of the pixel values of the reference pixels having an Rcomponent and the corresponding filter coefficients and significances ωby the sum total of the products of the filter coefficients and thesignificances ω corresponding to the reference pixels calculated at stepS128 and determines the quotient as a pixel value corresponding to thenoticed pixel of the image R′ which includes only smoothed R components.Meanwhile, also the smoothing section 82 divides the sum total of theproducts of the pixel values of the reference pixels having a Gcomponent and the corresponding filter coefficients and significances ωby the sum total of the products of the filter coefficients and thesignificances ω corresponding to the reference pixels calculated at stepS128 and determines the quotient as a pixel value corresponding to thenoticed pixel of the image G′ which includes only smoothed G components.

At step S130, the subtractor 83 subtracts the pixel value correspondingto the noticed pixel of the image G′, which only includes smoothed Gcomponents, from the smoothing section 82 from the pixel valuecorresponding to the noticed pixel of the image R′, which only includessmoothed R components, from the smoothing section 81, and determines thedifference as a pixel value of the noticed pixel of the color differenceimage C.

The processing returns to step S121 so that the processing at steps S121to 130 is repeated until it is discriminated at step S121 that allpixels have been used as a noticed pixel. When it is discriminated atstep S121 that all pixels have been used as a noticed pixel, the colordifference image production process is ended and the processing returnsto step S53 of FIG. 55.

It is to be noted that, since the process of the color difference imageproduction section 73 when it produces a color difference image D issimilar to the second process of the color difference image productionsection 72 when it produces the color difference image C describedabove, description of it is omitted.

In the second process for producing a color difference image C, since acontour of an object in an image is detected and smoothing is executedin parallel to the contour, occurrence of a color moire effect can besuppressed when compared with that in the first process for producingthe color difference image C.

Subsequently, a second example of a configuration of the imageprocessing section 7 which principally executes the second demosaicprocess is described with reference to FIG. 64. In the second example ofthe configuration of the image processing section 7, a color andsensitivity mosaic image from the image pickup system, color mosaicpattern information representative of a color mosaic arrangement of thecolor and sensitivity mosaic image and sensitivity mosaic patterninformation representative of a sensitivity mosaic arrangement of thecolor and sensitivity mosaic image are supplied to a sensitivityuniformization section 111.

The sensitivity uniformization section 111 performs a sensitivityuniformization process for the color and sensitivity mosaic image basedon the color mosaic pattern information and the sensitivity mosaicinformation and outputs a resulting color mosaic image M having auniformized sensitivity to the color interpolation section 52. It is tobe noted, however, that, since the color mosaic arrangement of theresulting color mosaic image M is not necessarily same as the colormosaic arrangement of the original color and sensitivity mosaic image,the sensitivity uniformization section 111 updates the color mosaicpattern information and supplies it to a color interpolation section112.

The color interpolation section 112 performs, similarly to the colorinterpolation section 52 of FIG. 45, a color interpolation process, inwhich the color mosaic pattern information is used, for the color mosaicimage M from the sensitivity uniformization section 111 to produceoutput images R, G and B.

FIG. 65 shows a first example of a configuration of the sensitivityuniformization section 111. The first example of the configuration is anexample of a configuration of the sensitivity uniformization section 111which executes the first sensitivity uniformization process in thesecond demosaic process described hereinabove with reference to FIGS.35, 41 and 42.

In the first example of the configuration of the sensitivityuniformization section 111, a color and sensitivity mosaic image fromthe image pickup system is supplied to a sensitivity compensationsection 121 and a validity discrimination section 123. Color mosaicpattern information is supplied to a missing interpolation section 124.Sensitivity mosaic pattern information is supplied to the sensitivitycompensation section 121 and the validity discrimination section 123.

The sensitivity compensation section 121 performs sensitivitycompensation for the color and sensitivity mosaic image based on arelative sensitivity value S obtained from a relative sensitivity valueLUT 122 and outputs the resulting color and sensitivity mosaic image tothe missing interpolation section 124. The relative sensitivity valueLUT 122 is a lookup table which outputs a relative sensitivity value Susing a sensitivity of a pixel as an index.

The validity discrimination section 123 compares the pixel value of eachof the pixels of the color and sensitivity mosaic image with thethreshold value θ_(H) of the saturation level and the threshold valueθ_(L) of the noise level to discriminate the validity of the pixel valueand supplies a result of the discrimination as discriminationinformation to the missing interpolation section 124. In thediscrimination information, information representative of “valid” or“invalid” regarding the pixel value of each pixel is described.

The missing interpolation section 124 uses, based on the discriminationinformation from the validity discrimination section 123, the pixelvalues of those pixels from among all pixels of thesensitivity-compensated color and sensitivity mosaic image whosediscrimination information is valid as they are, but uses, for each ofthose pixels whose discrimination information is invalid, the pixelvalues of those pixels having a color which is included most in thesensitivity-compensated color and sensitivity mosaic image tointerpolate the pixel value of the color component. Use of the pixelvalues of those pixels having a color which is included most in thismanner facilitates restoration of a high frequency component. Further,the missing interpolation section 124 updates the color mosaic patterninformation corresponding to the color mosaic arrangement of theproduced color mosaic image M and outputs the updated color mosaicpattern information to the color interpolation section 112.

Now, a second demosaic process executed principally by the secondexample of the configuration of the image processing section 7 shown inFIG. 64 is described. However, most part of the second demosaic processis similar to that of the first demosaic process described hereinabove.Therefore, a process different from that of the first demosaic processdescribed hereinabove, that is, a missing interpolation process of themissing interpolation section 124 which composes the sensitivityuniformization section 111 is described with reference to a flow chartof FIG. 66. In the following description, it is assumed that the numberof pixels having a G component is greatest in the color and sensitivitymosaic image. However, a similar process can be applied similarly alsowhere the number of pixels having any other color component is greatest.

At step S151, the missing interpolation section 124 discriminateswhether or not all pixels of the sensitivity-compensated color andsensitivity mosaic image have been used as a noticed pixel. If themissing interpolation section 124 discriminates that all pixels have notbeen used as a noticed pixel, then the processing advances to step S152.At step S152, the missing interpolation section 124 determines one byone pixel as a noticed pixel beginning with the left lowermost pixel andending with the right uppermost pixel of the sensitivity-compensatedcolor and sensitivity mosaic image.

At step S153, the missing interpolation section 124 discriminateswhether or not the discrimination information of the noticed pixel isinvalid. If the missing interpolation section 124 discriminates that thediscrimination information is invalid, then the processing advances tostep S154.

At step S154, the missing interpolation section 124 refers to the colormosaic pattern information to detect those pixels neighboring with thenoticed pixel (for example, 5×5 pixels centered at the noticed pixel)which have a G component and whose discrimination information is valid,and extracts the pixel values of the detected pixels (hereinafterreferred to as reference pixels). Further, the missing interpolationsection 124 acquires a number of filter coefficients set in advancecorresponding to relative positions of the reference pixels to thenoticed pixel, the number being equal to the number of the referencepixels. Furthermore, the missing interpolation section 124 multipliesthe pixel values of the reference pixels and the corresponding filtercoefficients and arithmetically operates the sum total of the products.Further, the missing interpolation section 124 divides the sum total ofthe products by the sum total of the used filter coefficients anddetermines the quotient as a pixel value of the noticed pixel of thecolor mosaic image M.

At step S155, the missing interpolation section 124 updates the color ofthe noticed pixel in the color mosaic pattern information to G.

It is to be noted that, if it is discriminated at step S153 that thediscrimination information of the noticed pixel is not invalid, then theprocesses at steps S154 and S155 are skipped.

The processing returns to step S151 so that the processing at steps S151to 155 is repeated until it is discriminated at step S151 that allpixels have been used as a noticed pixel. When it is discriminated atstep S151 that all pixels have been used as a noticed pixel, the missinginterpolation process is ended and the color mosaic image M obtained andthe updated color mosaic pattern information are supplied to the colorinterpolation section 112 in the following stage.

Now, a second example of a configuration of the sensitivityuniformization section 111 which can be used in place of the firstexample of the configuration of the sensitivity uniformization section111 shown in FIG. 65 is described with reference to FIG. 67.

The second example of the configuration is an example of a configurationfor allowing the sensitivity uniformization section 111 to execute thesecond sensitivity uniformization process of the second demosaic processdescribed hereinabove with reference to FIGS. 35, 43 and 44.

The following description proceeds assuming that, in the color andsensitivity mosaic image, the color of each pixel is one of the threeprimary colors of R, G and B as in the color and sensitivity mosaicpattern P10 of FIG. 14 or the color and sensitivity mosaic pattern P1 ofFIG. 15 and the sensitivity is one of sensitivities of four stages ofS0, S1, S2 and S3. However, the configuration and the operationdescribed below can be applied also to another color and sensitivitymosaic image which includes three colors other than R, G and B or afurther color and sensitivity mosaic image which includes four colors.Furthermore, they can be applied also to a color and sensitivity mosaicpattern wherein the number of stages of sensitivity is two or three.

In the second example of the configuration of the sensitivityuniformization section 111, a color and sensitivity mosaic image fromthe image pickup system, color mosaic pattern information andsensitivity mosaic pattern information are supplied to interpolationsections 132-1 to 132-4. The color mosaic pattern information issupplied also to an interpolation color determination section 131.

The interpolation color determination section 131 designates the color(interpolation color) of interpolation values to be interpolated by theinterpolation sections 132-1 to 132-3 based on the color mosaic patterninformation. Further, the interpolation color determination section 131updates the color mosaic pattern information in accordance withdetermination of the interpolation colors.

The interpolation section 131-1 performs an interpolation process of thesensitivity S0 for the color and sensitivity mosaic image in accordancewith the designation of an interpolation color from the interpolationcolor determination section 131 and outputs a resulting interpolationvalue corresponding to the sensitivity S0 to an adder 133. Theinterpolation section 131-2 performs an interpolation process of thesensitivity S1 for the color and sensitivity mosaic image in accordancewith the designation of the interpolation color from the interpolationcolor determination section 131 and outputs a resulting interpolationvalue corresponding to the sensitivity S1 to the adder 133. Theinterpolation section 131-3 performs an interpolation process of thesensitivity S2 for the color and sensitivity mosaic image in accordancewith the designation of the interpolation color from the interpolationcolor determination section 131 and outputs a resulting interpolationvalue corresponding to the sensitivity S2 to the adder 133. Theinterpolation section 131-4 performs an interpolation process of thesensitivity S3 for the color and sensitivity mosaic image in accordancewith the 131 designation of the interpolation color from theinterpolation color determination section and outputs a resultinginterpolation value corresponding to the sensitivity S3 to the adder133.

The adder 133 adds the interpolation values of the sensitivities S0 toS3 inputted thereto from the interpolation sections 132-1 to 132-4 foreach pixel and supplies the sum as a pixel value of a color mosaiccandidate image to a synthetic sensitivity compensation section 134.

The synthetic sensitivity compensation section 134 collates the pixelvalue of the color mosaic candidate image supplied thereto from theadder 133 with a synthetic sensitivity compensation LUT 135 and producesand supplies a color mosaic image wherein the resulting value is used asa pixel value to the color interpolation section 112. The syntheticsensitivity compensation LUT 135 allows a pixel value of the color andsensitivity mosaic image M using a pixel value of the color mosaiccandidate image as an index.

A second sensitivity uniformization process in the second demosaicprocess by the second example of the configuration of the sensitivityuniformization section 111 shown in FIG. 67 is described with referenceto a flow chart of FIG. 68.

At step S161, the interpolation sections 132-1 to 132-4 discriminatewhether or not all pixels of the color and sensitivity mosaic image havebeen used as a noticed pixel. If the interpolation sections 132-1 to132-4 discriminate that all pixels have not been used as a noticedpixel, then the processing advances to step S162. At step S162, theinterpolation sections 132-1 to 132-4 determine one by one pixel as anoticed pixel beginning with the left lowermost pixel and ending withthe right uppermost pixel of the color and sensitivity mosaic image.

At step S163, the interpolation color determination section 131 executesan interpolation color determination process based on the color mosaicpattern information and issues a notification of a resultinginterpolation color of the noticed pixel to the interpolation sections132-1 to 132-4.

Details of the interpolation color determination process of theinterpolation color determination section 131 are described withreference to a flow chart of FIG. 69. It is to be noted that the objectof the interpolation color determination process is to interpolate thepixel value of the noticed pixel using pixels comparatively neighboringwith the noticed pixel and it is assumed that the color mosaicarrangement of the color and sensitivity mosaic image has a Bayerarrangement.

At step S171, the interpolation color determination section 131 refersto the color mosaic pattern information to discriminate the color of thenoticed pixel.

If it is discriminated at step S171 that the color of the noticed pixelis G, then the processing advances to step S172. In this instance, alsothe colors of the four pixels neighboring in the oblique directions withthe noticed pixel are G. At step S172, the interpolation colordetermination section 131 determines the interpolation color of thenoticed pixel as G and issues a notification of this to theinterpolation sections 132-1 to 132-4. Further, the interpolation colordetermination section 131 updates the color mosaic pattern informationcorresponding to the noticed pixel to G.

If it is discriminated at step S171 that the color of the noticed pixelis R, then the processing advances to step S173. In this instance, thecolors of the four pixels neighboring in the oblique directions with thenoticed pixel are B. At step S173, the interpolation color determinationsection 131 determines the interpolation color of the noticed pixel as Band issues a notification of this to the interpolation sections 132-1 to132-4. Further, the interpolation color determination section 131updates the color mosaic pattern information corresponding to thenoticed pixel to G.

If it is discriminated at step S171 that the color of the noticed pixelis B, then the processing advances to step S174. In this instance, alsothe colors of the four pixels neighboring in the oblique directions withthe noticed pixel are R. At step S174, the interpolation colordetermination section 131 determines the interpolation color of thenoticed pixel as R and issues a notification of this to theinterpolation sections 132-1 to 132-4. Further, the interpolation colordetermination section 131 updates the color mosaic pattern informationcorresponding to the noticed pixel to R.

With the interpolation color determination process described above, theinterpolation color of the noticed pixel is designated so that R and Bof the color and sensitivity mosaic image whose color mosaic arrangementis a Bayer arrangement are exchanged for each other. Therefore, also theupdated color mosaic pattern information maintains the Bayerarrangement.

The processing returns to step S164 of FIG. 68. At step S164, theinterpolation sections 132-1 to 132-4 individually perform aninterpolation process for the color and sensitivity mosaic image inaccordance with the designation of the interpolation color from theinterpolation color determination section 131 to produce aninterpolation value corresponding to the sensitivity S0, S1, S2 or S3and outputs the interpolation value to the adder 133.

More particularly, for example, the interpolation section 132-1 detects,from among pixels positioned in the neighborhood of the noticed pixel ofthe color and sensitivity mosaic image (for example, from among 5×5pixels centered at the noticed pixel), those pixels which have the colordesignated from the interpolation color determination section 131 andwhose sensitivity is S0, and extracts the pixel values of the detectedpixels (hereinafter referred to as reference pixels). Further, theinterpolation section 132-1 acquires a number of filter coefficients setin advance corresponding to relative positions of the detected referencepixels to the noticed pixel, the number being equal to the number of thereference pixels. Furthermore, the interpolation section 132-1multiplies the pixel values of the reference pixels and thecorresponding filter coefficients and arithmetically operates the sumtotal of the products. Further, the interpolation section 132-1 dividesthe sum total of the products by the sum total of the used filtercoefficients and determines the quotient as an interpolation valuecorresponding to the sensitivity S0 of the noticed pixel.

It is to be noted that the interpolation processes for the sensitivitiesS1 to S3 by the interpolation sections 132-2 to 132-3 are similar to theinterpolation process for the sensitivity S0 by the interpolationsection 132-1, and therefore, description of it is omitted.

At step S165, the adder 133 adds the interpolation values for thesensitivities S0 to S3 corresponding to the noticed pixel inputtedtherefrom from the interpolation sections 132-1 to 132-4 and suppliesthe sum as a pixel value of the color mosaic candidate imagecorresponding to the noticed pixel to the synthetic sensitivitycompensation section 133.

At step S166, the synthetic sensitivity compensation section 134collates the pixel value of the color mosaic candidate image suppliedthereto from the adder 133 with the synthetic sensitivity compensationLUT 135 and determines a resulting value as a pixel value of the colormosaic image M corresponding to the noticed pixel.

The processing returns to step S161 so that the processing at steps S161to 166 is repeated until it is discriminated at step S161 that allpixels have been used as a noticed pixel. When it is discriminated atstep S161 that all pixels have been used as a noticed pixel, the secondsensitivity uniformization process in the second demosaic process isended.

It is to be noted that, although a color interpolation process isperformed by the color interpolation section 112 for the color mosaicimage M obtained by the second sensitivity uniformization process of thesecond demosaic process, since the process is similar to the colorinterpolation process described hereinabove with reference to the flowchart of FIG. 55, description of it is omitted.

FIG. 70 illustrates an outline of a third demosaic process of the imageprocessing system which includes the image processing section 7 as aprincipal component.

The third demosaic process includes, as seen in FIG. 70, aby-sensitivity-basis color interpolation process wherein RGB componentsof pixels of a color and sensitivity mosaic image obtained by processingof the image pickup section are interpolated without changing thesensitivities of the pixels to produce a sensitivity mosaic image MsRfor an R component, a sensitivity mosaic image MsG for a G component anda sensitivity mosaic image MsB for a B component, and a sensitivityuniformization process for uniformizing the sensitivities of thesensitivity mosaic image for an R component, the sensitivity mosaicimage for a G component and the sensitivity mosaic image for a Bcomponent to produce output images R, G and B, respectively.

The by-sensitivity-basis color interpolation process of the thirddemosaic process includes an extraction process for extracting onlythose pixels which have the same sensitivity from the color andsensitivity mosaic image, a color interpolation process forinterpolating the pixel values of the RGB components of the pixelsextracted by the extraction process, and an insertion process forsynthesizing the pixel values interpolated by the color interpolationprocess for each of the RGB components to produce sensitivity mosaicimages.

For example, in the extraction process, only the pixels which have thesensitivity S1 are extracted from the color and sensitivity mosaic imageto produce a color mosaic image McS1 wherein the pixels are disposed ina checkered manner. In the color interpolation process, an image Rs1wherein the pixels which have the sensitivity S1 and have an R componentare disposed in a checkered manner, another image Gsl wherein the pixelswhich have the sensitivity S1 and have a G component are disposed in acheckered manner and a further image Bs1 wherein the pixels which havethe sensitivity S1 and have a B component are disposed in a checkeredmanner are produced from the color mosaic image McS1.

For example, in the insertion process, an image RS0 and another imageRS1 produced by the color interpolation process are combined to producea sensitivity mosaic image MsR.

Subsequently, a third example of a configuration of the image processingsection 7 which principally executes the third demosaic process isdescribed with reference to FIG. 73.

In the third example of the configuration of the image processingsection 7, a color and sensitivity mosaic image from the image pickupsystem is supplied to a by-sensitivity-basis color interpolation section151. Color mosaic pattern information representative of a color mosaicarrangement of the color and sensitivity mosaic image is supplied to theby-sensitivity-basis color interpolation section 151. Sensitivity mosaicpattern information representative of a sensitivity mosaic arrangementof the color and sensitivity mosaic image is supplied to theby-sensitivity-basis color interpolation section 151 and sensitivityuniformization sections 152 to 154.

It is to be noted that, in the following description, unless otherwisespecified, the color and sensitivity mosaic image has the color andsensitivity mosaic pattern P3 of FIG. 7. In particular, each pixel has acolor which is one of the three primary colors of R, G and B and has asensitivity of one of S0 and S1. Further, where attention is paid toonly the pixels of the sensitivity S0 irrespective of the color, theyare arranged in a checkered manner. Similarly, the pixels of thesensitivity S1 are arranged in a checkered manner.

However, the configuration and the operation described below can beapplied also to another color and sensitivity mosaic image having threecolors other than R, G and B or a further color and sensitivity mosaicimage which has four colors.

The by-sensitivity-basis color interpolation section 151 performs aby-sensitivity-basis color interpolation process for the color andsensitivity mosaic image and supplies resulting sensitivity mosaic imageMsR for an R component, sensitivity mosaic image MsG for a G componentand sensitivity mosaic image MsB for a B component to corresponding onesof the sensitivity uniformization sections 152 to 154, respectively.

The sensitivity uniformization section 152 performs a sensitivityuniformization process for the sensitivity mosaic image MsR for an Rcomponent to produce an output image R. The sensitivity uniformizationsection 153 performs a sensitivity uniformization process for thesensitivity mosaic image MsG for a G component to produce an outputimage G. The sensitivity uniformization section 154 performs asensitivity uniformization process for the sensitivity mosaic image MsBfor a B component to produce an output image B.

FIG. 74 shows an example of a configuration of the by-sensitivity-basiscolor interpolation section 151. In the by-sensitivity-basis colorinterpolation section 151, the color and sensitivity mosaic image, colormosaic pattern information and sensitivity mosaic pattern informationare supplied to an extraction section 161.

The extraction section 161 performs an extraction process of thesensitivity S1 (in the present case, i=0 or 1) for the color andsensitivity mosaic image and supplies a resulting color mosaic imageMcSi which includes pixels of the sensitivity Si to a colorinterpolation section 162. It is to be noted that the color mosaic imageMcSi is an image represented using an st coordinate system differentfrom the xy coordinate system of the original color and sensitivitymosaic image (details are hereinafter described with reference to FIGS.78 and 79). Further, the extraction section 161 produces color mosaicpattern information of the sensitivity Si representative of a colormosaic arrangement of the color mosaic image McSi and supplies the colormosaic pattern information to the color interpolation section 162.Furthermore, the extraction section 161 produces original positioninformation of the sensitivity Si which has a positional relationshipbetween the color mosaic image McSi and the original color andsensitivity mosaic image and supplies the original position informationof the sensitivity Si to insertion sections 163 to 165.

The color interpolation section 162 interpolates RGB components of allpixels of the color mosaic image McSi from the extraction section 161and supplies resulting images Rsi, Gsi and Bsi to the correspondinginsertion sections 163 to 165, respectively. The image Rsi is an imagecomposed of pixel values of R components corresponding to the pixels ofthe color mosaic image McSi. The image Gsi is an image composed of pixelvalues of G components corresponding to the pixels of the color mosaicimage McSi. The image Bsi is an image composed of pixel values of Bcomponents corresponding to the pixels of the color mosaic image McSi.Further, the images Rsi, Gsi and Bsi are represented using a coordinatesystem same as that of the color mosaic image McSi. It is to be notedthat the color interpolation section 162 is configured in a similarmanner as in the example of the configuration of the color interpolationsection 52 shown in FIG. 47.

The insertion section 163 combines a number of images Rsi of an Rcomponent equal to the number of kinds of sensitivities supplied fromthe color interpolation section 162 based on the original positioninformation of the sensitivity Si supplied from the extraction section161 to produce a sensitivity mosaic image MsR, and supplies thesensitivity mosaic image MsR to the sensitivity uniformization section152. The insertion section 164 combines a number of images Gsi of a Gcomponent equal to the number of kinds of sensitivities supplied fromthe color interpolation section 162 based on the original positioninformation of the sensitivity Si supplied from the extraction section161 to produce a sensitivity mosaic image MsG, and supplies thesensitivity mosaic image MsG to the sensitivity uniformization section153. The insertion section 165 combines a number of images Bsi of a Bcomponent equal to the number of kinds of sensitivities supplied fromthe color interpolation section 162 based on the original positioninformation of the sensitivity Si supplied from the extraction section161 to produce a sensitivity mosaic image MsB, and supplies thesensitivity mosaic image MsB to the sensitivity uniformization section154.

FIG. 75 shows an example of a configuration of the sensitivityuniformization section 152. In the sensitivity uniformization section152, the sensitivity mosaic image MsR supplied from the insertionsection 163 of the by-sensitivity-basis color interpolation section 151is supplied to a local sum calculation section 171. The local sumcalculation section 171 performs, for each pixel of the sensitivitymosaic image MsR, a local sum calculation process using pixelsneighboring with the pixel and supplies resulting the local sumcorresponding to each of the pixels to a synthetic sensitivitycompensation section 172. The synthetic sensitivity compensation section172 collates the local sums with a synthetic sensitivity compensationLUT 173 to acquire corresponding compensation values and produces anoutput image R using the compensation values as pixel values. Thesynthetic sensitivity compensation LUT 173 can supply a correspondingcompensation value when a local sum is inputted as an index thereto.

It is to be noted that examples of configurations of the sensitivityuniformization sections 153 and 154 are similar to the example of theconfiguration of the sensitivity uniformization section 152 shown inFIG. 75, and therefore, description of them is omitted.

Subsequently, a third demosaic process by the third example of theconfiguration of the image processing section 7 shown in FIG. 73 isdescribed with reference to a flow chart of FIG. 76.

At step 181, the by-sensitivity-basis color interpolation section 151performs a by-sensitivity-basis color interpolation process for thecolor and sensitivity mosaic image to produce an R component sensitivitymosaic image MsR, a G component sensitivity mosaic image MsG and a Bcomponent sensitivity mosaic image MsB and supplies them to thesensitivity uniformization sections 152 to 154, respectively.

Details of the by-sensitivity-basis color interpolation process of theby-sensitivity-basis color interpolation section 151 are described withreference to a flow chart of FIG. 77. At step S191, the extractionsection 161 discriminates whether or not all sensitivities (in thepresent case, S0 and S1) included in the sensitivity mosaic patterninformation have been designated. If the extraction section 161discriminates that all sensitivities have not been designated, then theprocessing advances to step S192.

At step S192, the extraction section 161 determines one of all kinds ofsensitivities included in the sensitivity mosaic pattern information.The designated sensitivity is represented by Si.

At step S193, the extraction section 161 extracts only pixels of thesensitivity Si from among all pixels of the color and sensitivity mosaicimage to produce a color mosaic image McSi of the sensitivity Si andsupplies the color mosaic image McSi to the color interpolation section162. Further, the extraction section 161 produces original positioninformation of the sensitivity Si which keeps a positional relationshipbetween the color mosaic image McSi and the original color andsensitivity mosaic image and supplies the original position informationto the insertion sections 163 to 165. Further, the extraction section161 produces color mosaic pattern information of the sensitivity Sirepresentative of a color mosaic arrangement of the color mosaic imageMcSi and supplies the color mosaic pattern information to the colorinterpolation section 162.

Details of the process at step S193 are described with reference toFIGS. 78 and 79.

Since pixels of the sensitivity Si extracted do not have a pixeldistance of the original color and sensitivity mosaic image, the colormosaic image McSi of the sensitivity Si produced is formed in a gratingwherein the pixel distance, the original and the direction are differentfrom those of the original color and sensitivity mosaic image.Therefore, the extraction section 61 produces, simultaneously withproduction of the color mosaic image McSi, original position informationwhich allows, for each pixel, information of the original position to bereferred to based on a corresponding relationship between the coordinatesystem of the original color and sensitivity mosaic image and thecoordinate system of the color mosaic image McSi.

The corresponding relationship between the coordinate systems of theoriginal color and sensitivity mosaic image and the color mosaic imageMcSi to be produced is such as illustrated in FIG. 78 or 79. Referringto FIGS. 78 and 79, the original color and sensitivity mosaic image isindicated on the xy coordinate system while the color mosaic image McSiis indicated on the st coordinate system. Further, ▪ of the color andsensitivity mosaic image represents a pixel of the sensitivity S0, and □of the color and sensitivity mosaic image represents a pixel of thesensitivity S0. By using the st coordinate system set obliquely withrespect to the xy coordinate system in this manner, pixels of thesensitivity Si disposed in a checkered manner on the original color andsensitivity mosaic image can be extracted as a pixel arrangement of anequal distance grating.

Extraction of pixels of the sensitivity S0 represented by ▪ of the colorand sensitivity mosaic image is described with reference to FIG. 78. Forexample, a pixel A in FIG. 78 is represented as (x_(A), y_(A)) on the xycoordinate system which represents the original color and sensitivitymosaic image but is represented as (s_(A), t_(A)) on the st coordinatesystem which represents the color mosaic image McSi to be produced.(s_(A), t_(A)) and (x_(A), y_(A)) have such relationships as representedby the following expression (22):s_(A)={(x_(A)−1)+y_(A)}/2t_(A)={(x_(max)−1−x_(A))+y_(A)}/2   (22)

The extraction section 161 applies the coordinates (x_(A), y_(A)) of thepixel of the sensitivity S0 of the original color and sensitivity mosaicimage to the expression (22) to calculate the coordinates (s_(A), t_(A))on the color mosaic image McSi and uses the value of the pixel for thecoordinates to produce a color mosaic image McSi. Simultaneously, theextraction section 161 places the coordinates (x_(A), y_(A)) in acorresponding relationship to the coordinates (s_(A), t_(A)) into theoriginal position information of the sensitivity S0.

Extraction of a pixel of the sensitivity S1 represented by □ of thecolor and sensitivity mosaic image is described with reference to FIG.79. For example, a pixel B in FIG. 79 is represented as (x_(B), y_(B))on the xy coordinate system which represents the original color andsensitivity mosaic image but is represented as (s_(B), t_(B)) on the stcoordinate system which represents the color mosaic image McSi to beproduced. (s_(B), t_(B)) and (x_(B), y_(B)) have such a relationship asrepresented by the following expression (23):s_(B)=(x_(B)+y_(B))/2t_(B)={(x_(max)−1−x_(B))+y_(B)}/2   (23)

The extraction section 161 applies the coordinates (x_(B), y_(B)) of thepixel of the sensitivity S1 of the original color and sensitivity mosaicimage to the expression (22) to calculate the coordinates (s_(B), t_(B))on the color mosaic image McSi and uses the value of the pixel for thecoordinates to produce a color mosaic image McSi. Simultaneously, theextraction section 161 places the coordinates (x_(B), y_(B)) in acorresponding relationship to the coordinates (s_(B), t_(B)) into theoriginal position information of the sensitivity S1.

Referring back to FIG. 77, the color interpolation section 162interpolates RGB components of all pixels of the color mosaic image McSifrom the extraction section 161 to produce images Rsi, Gsi and Bsi andsupplies the images Rsi, Gsi and Bsi to the corresponding insertionsections 163 to 165, respectively. It is to be noted that details ofprocessing of the color interpolation section 162 are similar to thoseof the color interpolation process of the color interpolation section 52described with reference to FIG. 55, and therefore, description of themis omitted.

The processing returns to step S191 so that the processing at steps S191to S194 is repeated until it is discriminated at step S191 that allsensitivities included in the sensitivity mosaic pattern informationhave been designated. When it is discriminated at step S191 that allsensitivities included in the sensitivity mosaic pattern informationhave been designated, the processing advances to step S195.

At step S195, the insertion section 163 combines a number of images Rsiof an R component (in the present case, the images Rs0 and images Rs1)equal to the number of kinds of sensitivities supplied from the colorinterpolation section 162 based on all of the original positioninformation supplied from the extraction section 161 to produce asensitivity mosaic image MsR, and supplies the sensitivity mosaic imageMsR to the sensitivity uniformization section 152. Similarly, theinsertion section 164 produces and supplies a sensitivity mosaic imageMsG to the sensitivity uniformization section 153, and the insertionsection 165 produces and supplies a sensitivity mosaic image MsB to thesensitivity uniformization section 154.

The processing returns to step S182 of FIG. 76. At step S182, thesensitivity uniformization section 152 performs a sensitivityuniformization process for the R component sensitivity mosaic image MsRto produce an output image R. The sensitivity uniformization section 153performs a sensitivity uniformization process for the G componentsensitivity mosaic image MsG to produce an output image G. Thesensitivity uniformization section 154 performs a sensitivityuniformization for the B component sensitivity mosaic image MsB toproduce an output image B.

The sensitivity uniformization process of the sensitivity uniformizationsection 152 is described with reference to a flow chart of FIG. 80. Atstep S201, the local sum calculation section 171 discriminates whetheror not all pixels of the R component sensitivity mosaic image MsR havebeen used as a noticed pixel. If the local sum calculation section 171discriminates that all pixels have not been used as a noticed pixel,then the processing advances to step S202. At step S202, the local sumcalculation section 171 determines one by one pixel as a noticed pixelbeginning with the left lowermost pixel and ending with the rightuppermost pixel of the sensitivity mosaic image MsR.

At step S203, the local sum calculation section 171 calculates a localsum corresponding to the noticed pixel and supplies it to the syntheticsensitivity compensation section 172. More particularly, the pixelvalues of 5×5 pixels (hereinafter referred to as reference pixels)centered at the noticed pixel are extracted, and the pixel values aremultiplied by such filter coefficients set in advance corresponding torelative positions of the reference pixels to the noticed pixel as seenin FIG. 81, whereafter the sum total of the products is arithmeticallyoperated. Further, the sum total of the products is divided by the sumtotal of the 25 filter coefficients, and the quotient is determined as alocal sum corresponding to the noticed pixel.

At step S204, the synthetic sensitivity compensation section 172collates the local sum with the synthetic sensitivity compensation LUT173 to acquire a corresponding compensation value and determines thecompensation value as a pixel value of the output image R correspondingto the noticed pixels.

The processing returns to step S201 so that the processing at steps S201to S204 is repeated until it is discriminated at step S201 that allpixels have been used as a noticed pixel. When it is discriminated atstep S201 that all pixels have been used as a noticed pixel, thesensitivity uniformization process is ended, and the processing returnsto FIG. 76.

It is to be noted that, although also the sensitivity uniformizationsections 153 and 154 execute a similar sensitivity uniformizationprocess in parallel to the sensitivity uniformization process of thesensitivity uniformization section 152, detailed description of it isomitted.

Description of the third demosaic process by the third example of theconfiguration of the image processing section 7 is ended therewith.

Subsequently, an outline of a fourth demosaic process of the imageprocessing system including the image processing section 7 as aprincipal component is described.

The fourth demosaic process includes a luminance image productionprocess for producing a luminance image from a color and sensitivitymosaic image obtained by processing of the image pickup system, and amonochromatic image process for producing output images R, G and B usingthe color and sensitivity mosaic image and the luminance image.

FIG. 82 shows a fourth example of a configuration of the imageprocessing section 7 which principally executes the fourth demosaicprocess.

In the fourth example of the configuration of the image processingsection 7, a color and sensitivity mosaic image from the image pickupsystem, color mosaic pattern information which indicates a color mosaicarrangement of the color and sensitivity mosaic image and sensitivitymosaic pattern information which indicates a sensitivity mosaicarrangement of the color and sensitivity mosaic image are supplied to aluminance image production section 181 and monochromatic imageproduction sections 182 to 184.

It is to be noted that, in the following description, unless otherwisespecified, the color and sensitivity mosaic image has the color andsensitivity mosaic pattern P2 of FIG. 6. In particular, each pixel has acolor which is one of the three primary colors of R, G and B and has asensitivity of one of S0 and S1, and further, where attention is paidonly to the color irrespective of the sensitivity, the pixels of thecolor are arranged in a Bayer arrangement.

However, the configuration and the operation described below can beapplied also to another color and sensitivity mosaic image whichincludes three colors other than R, G and B or a further color andsensitivity mosaic image which includes four colors.

The luminance image production section 181 performs a luminance imageproduction process for the color and sensitivity mosaic image suppliedthereto and supplies a resulting luminance image to the monochromaticimage production sections 182 to 184.

The monochromatic image production section 182 produces an output imageR using the color and sensitivity mosaic image and the luminance imagesupplied thereto. The monochromatic image production section 183produces an output image G using the color and sensitivity mosaic imageand the luminance image supplied thereto. The monochromatic imageproduction section 184 produces an output image B using the color andsensitivity mosaic image and luminance image supplied thereto.

FIG. 83 shows a first example of a configuration of the luminance imageproduction section 181. In the first example of the configuration of theluminance image production section 181, a color and sensitivity mosaicimage, color mosaic pattern information and sensitivity mosaic patterninformation are supplied to estimation sections 191 to 193.

The estimation section 191 performs an R component estimation processfor the color and sensitivity mosaic image and supplies an estimationvalue R′ of an R component for each pixel obtained by the process to amultiplier 194. The estimation section 192 performs a G componentestimation process for the color and sensitivity mosaic image andsupplies an estimation value G′ of a G component for each pixel obtainedby the process to another multiplier 195. The estimation section 193performs a B component estimation process for the color and sensitivitymosaic image and supplies an estimation value B′ of a B component foreach pixel obtained by the process to a further multiplier 196.

The multiplier 194 multiplies the estimation value R′ supplied from theestimation section 191 by a color balance coefficient K_(R) and outputsthe product to an adder 197. The multiplier 195 multiplies theestimation value G′ supplied from the estimation section 192 by a colorbalance coefficient K_(G) and outputs the product to the adder 197. Themultiplier 196 multiplies the estimation value B′ supplied from theestimation section 193 by a color balance coefficient K_(B) and outputsthe product to the adder 197.

The adder 197 adds the product R′□k_(R)inputted from the multiplier 194,the product G′□k_(G)inputted from the multiplier 195 and the productB′□k_(B)inputted from the multiplier 196, and produces a luminancecandidate image wherein the resulting sum is used as a pixel value andsupplies the luminance candidate image to a noise removal section 198.

Here, the color balance coefficients k_(R), k_(G) and k_(B) are valuesset in advance and, for example, k_(R)=0.3, k_(G)=0.6 and k_(B)=0.1. Itis to be noted that, basically, the color balance coefficients k_(R),k_(G) and k_(B) may have any values only if they can be used tocalculate, as a luminance candidate value, a value having a correlationto a luminance variation. Accordingly, for example, the color balancecoefficients may be k_(R)=k_(G)=k_(B).

The noise removal section 198 performs a noise removal process for theluminance candidate image supplied from the adder 197 and supplies theresulting luminance image to monochromatic image production sections 182to 184.

FIG. 84 shows an example of a configuration of the monochromatic imageproduction section 182. In the monochromatic image production section182, the color and sensitivity mosaic image, the color mosaic patterninformation and the sensitivity mosaic pattern information are suppliedto an interpolation section 201. The luminance image is supplied to aratio value calculation section 202 and a multiplier 203.

The interpolation section 201 performs an interpolation process for thecolor and sensitivity mosaic image and outputs an R candidate imagewherein all resulting pixels have pixel values of an R component to theratio value calculation section 202. The ratio value calculation section202 calculates a low-frequency component of an intensity ratio (thelow-frequency component is hereinafter referred to merely as anintensity ratio) between corresponding pixels of the R candidate imageand the luminance image and produces ratio value information whichrepresents an intensity ratio corresponding to each pixel, and suppliesthe ratio value information to the multiplier 203.

The multiplier 203 multiplies the pixel value of each pixel of theluminance image by the corresponding intensity ratio and produces anoutput image R having the product as a pixel value.

It is to be noted that, since also examples of a configuration of themonochromatic image production sections 183 and 184 are similar to theexample of the configuration of the monochromatic image productionsection 182, description of them is omitted.

Now, the fourth demosaic process by the fourth example of theconfiguration of the image processing section 7 is described withreference to a flow chart of FIG. 85.

At step S211, the luminance image production section 181 performs aluminance image production process for the color and sensitivity mosaicimage to produce a luminance image and supplies the luminance image tothe monochromatic image production sections 182 to 184.

The luminance image production process of the luminance image productionsection 181 is described with reference to a flow chart of FIG. 86.

At step S221, the estimation sections 191 to 193 discriminate whether ornot all pixels of the color and sensitivity mosaic image have been usedas a noticed pixel. If the estimation sections 191 to 193 discriminatethat all pixels have not been used as a noticed pixel, then theprocessing advances to step S222. At step S222, the estimation sections191 to 193 determine one by one pixel as a noticed pixel beginning withthe left lowermost pixel and ending with the right uppermost pixel ofthe color and sensitivity mosaic image.

At step S223, the estimation section 191 performs an R componentestimation process for the color and sensitivity mosaic image toestimate an estimation value R′ corresponding to the noticed pixel andsupplies the estimation value R′ to the multiplier 194. The estimationsection 192 performs a G component estimation process for the color andsensitivity mosaic image to estimate an estimation value G′corresponding to the noticed pixel and supplies the estimation value G′to the multiplier 194. The estimation section 193 performs a B componentestimation process for the color and sensitivity mosaic image toestimate an estimation value B′ corresponding to the noticed pixel andsupplies the estimation value B′ to the multiplier 194.

The R component estimation process of the estimation section 191 isdescribed with reference to a flow chart of FIG. 87. At step S231, theestimation section 191 refers to the color mosaic pattern informationand the sensitivity mosaic pattern information to detect those of pixelsneighboring with the noticed pixel (for example, 15×15 pixels centeredat the noticed pixel) which have an R component and have the sensitivityS0, and extracts the pixel values of the detected pixels (hereinafterreferred to as reference pixels).

At step S232, the estimation section 191 acquires a number of such Rcomponent interpolation filter coefficients set in advance correspondingto relative positions of the reference pixels to the noticed pixel asshown in FIG. 88, the number being equal to the number of the referencepixels. Further, the estimation section 191 multiplies the pixel valuesof the reference pixels and the corresponding filter coefficients andarithmetically operates the sum total of the products. Furthermore, theestimation section 191 divides the sum total of the products by the sumtotal of the used R component interpolation filter coefficients toacquire a first quotient.

At step S233, the estimation section 191 refers to the color mosaicpattern information and the sensitivity mosaic pattern information todetect those of pixels neighboring with the noticed pixel (for example,15×15 pixels centered at the noticed pixel) which have an R componentand have the sensitivity S1, and extracts the pixel values of thedetected pixels (hereinafter referred to as reference pixels).

At step S234, the estimation section 191 acquires a number of Rcomponent interpolation filter coefficients corresponding to relativepositions of the reference pixels to the noticed pixel, the number beingequal to the number of the reference pixels. Further, the estimationsection 191 multiplies the pixel values of the reference pixels and thecorresponding filter coefficients and arithmetically operates the sumtotal of the products. Furthermore, the estimation section 191 dividesthe sum total of the products by the sum total of the used interpolationfilter coefficients to acquire a second quotient.

At step S235, the estimation section 191 adds the first quotientacquired at step S232 and the second quotient acquired at step S234. Atstep S235, the estimation section 191 collates the sum of the firstquotient and the second quotient arithmetically operated at step S235with a synthetic sensitivity compensation LUT (hereinafter described)built therein to acquire a compensation value of a compensatedsensitivity characteristic. The acquired compensation value isdetermined as an estimation value R′ corresponding to the noticed pixel.The processing returns to step S224 of FIG. 86.

It is to be noted that, since the G component interpolation processes ofthe estimation section 192 and the B component interpolation processesof the estimation section 193 are similar to the R componentinterpolation process of the estimation section 191, description of themis omitted. It is to be noted, however, in the G component estimationprocess of the estimation section 192, reference pixels are detectedfrom among 7×7 pixels centered at the noticed pixel, and further, the Gcomponent interpolation filter coefficients illustrated in FIG. 89 areused.

Here, the synthetic sensitivity compensation LUT used by the estimationsection 191 is described with reference to FIGS. 90 to 92. FIG. 90 showsa characteristic curve b of pixels of the sensitivity S0 and anothercharacteristic curve a of pixels of the sensitivity S1, and the axis ofabscissa indicates the intensity of incoming light and the axis ofordinate indicate the pixel value. In FIG. 90, the sensitivity S1 of thehigh sensitivity has a sensitivity as high as four times that of thesensitivity S0 of the low sensitivity.

In the estimation process, a first quotient calculated from a pixel ofthe sensitivity S0 measured with such a characteristic as indicated bythe characteristic curve b of FIG. 90 and a second quotient calculatedusing a pixel of the sensitivity S1 measured with such a characteristicas indicated by the characteristic curve a of FIG. 90 are added.Accordingly, the sum of the first quotient and the second quotient hassuch a characteristic synthesized from the characteristics of thesensitivity S0 and the sensitivity S1 as indicated by a characteristiccurve c of FIG. 91.

While the synthesized characteristic curve c exhibits a characteristicof a wide dynamic range from a low luminance to a high luminance, sinceit has a shape of a polygonal line, an original linear characteristic isrestored using a characteristic curve reverse to the sensitivitycharacteristic curve c. More particularly, the sum of the first productand the second product is applied to a reverse characteristic curve d tothe sensitivity characteristic curve c of FIG. 91 as shown in FIG. 92 tocompensate for the non-linearity.

In particular, the synthetic sensitivity compensation LUT is obtained byconverting the reverse characteristic curve d of FIG. 92 into a lookuptable.

Description is given with reference back to FIG. 86. At step S224, themultiplier 194 multiplies the estimation value R′ supplied from theestimation section 191 by a color balance coefficient k_(R) and outputsthe product to the adder 197. The multiplier 195 multiplies theestimation value G′ supplied from the estimation section 192 by a colorbalance coefficient k_(G) and outputs the product to the adder 197. Themultiplier 196 multiplies the estimation value B′ supplied from theestimation section 193 by a color balance coefficient k_(B) and outputsthe product to the adder 197. The adder 197 adds the product R′·k_(R)inputted from the multiplier 194, the product G′·k_(G) inputted from themultiplier 195 and the product B′·k_(B) inputted from the multiplier196, and determines the sum as a pixel value (luminance candidate value)of a luminance candidate image corresponding to the noticed pixel.

The processing returns to step S221 so that the processing at steps S221to S224 is repeated until it is discriminated at step S221 that allpixels have been used as a noticed pixel. When it is discriminated atstep S221 that all pixels have been used as a noticed pixel, theprocessing advances to step S225. It is to be noted that the luminancecandidate image produced by the processes at steps S221 to 224 issupplied to the noise removal section 198.

At step S225, the noise removal section 198 performs a noise removalprocess for the luminance candidate image supplied thereto from theadder 197 to produce a luminance image and supplies the luminance imageto the monochromatic image production sections 182 to 184.

The noise removal process of the noise removal section 198 is describedwith reference to a flow chart of FIG. 93. At step S241, the noiseremoval section 198 discriminates whether or not all pixels of theluminance candidate image have been used as a noticed pixel. If thenoise removal section 198 discriminates that all pixels have not beenused as a noticed pixel, then the processing advances to step S242. Atstep S242, the noise removal section 198 determines one by one pixel asa noticed pixel beginning with the left lowermost pixel and ending withthe right uppermost pixel of the luminance candidate image.

At step S243, the noise removal section 198 acquires the pixel values(luminance candidate values) of the pixels positioned upwardly,downwardly, leftwardly and rightwardly of the noticed pixel andsubstitutes the acquired luminance candidate values of the pixelspositioned upwardly, downwardly, leftwardly and rightwardly of thenoticed pixel into variables a3, a0, a1 and a2, respectively.

At step S244, the noise removal section 198 executes a directionselective smoothing process to acquire a smoothed value corresponding tothe noticed pixel.

The direction selective smoothing process of the noise removal section198 is described with reference to a flow chart of FIG. 94. At stepS251, the noise removal section 198 applies the variables a3, a0, a1, a2to the following expression (24) to calculate a luminance gradientvector g corresponding to the noticed pixel:

luminance gradient vector g=(a2−a1,a3−a0)   (24)

At step S252, the noise removal section 198 arithmetically operates themagnitude (absolute value)

∥∇∥ of the luminance gradient vector g.

At step S253, the noise removal section 198 applies the variables a0 toa3 to the following expressions (25) and (26) to calculate a smoothedcomponent Hh in the horizontal direction and a smoothed component Hv inthe vertical direction corresponding to the noticed pixel:Hh=(a1+a2)/2   (25)Hv=(a3+a0)/2   (26)

At step S254, the noise removal section 198 arithmetically operates asignificance wh in the horizontal direction and a significance wv in thevertical direction corresponding to the absolute value

∥g∥ of the luminance gradient vector g.

More particularly, where the absolute value ∥g∥ of the luminancegradient vector g is higher than 0, the absolute value of the innerproduct of the normalized luminance gradient vector g/∥g∥ and the vector(1, 0) is subtracted from 1 to obtain the significance wh in thehorizontal direction as given by the following expression (27). Further,the absolute value of the inner product of the normalized luminancegradient vector g/∥g∥ and the vector (0, 1) is subtracted from 1 toobtain the significance wv in the vertical direction as given by thefollowing expression (28).wh=1−|(g/∥g∥,(1,0))|  (27)wv=1−|(g/∥g∥,(0,1))|  (28)

Where the absolute value ∥g∥ of the luminance gradient vector g is 0,the smoothing contribution rate wh in the horizontal direction and thesmoothing contribution rate wv in the vertical direction are both set to0.5.

At step S255, the noise removal section 198 arithmetically operates asmoothed value a corresponding to the noticed pixel using the followingexpression (29):α=(wh·Hh+wv·Hv)/(wh+wv)   (29)

The processing returns to step S245 of FIG. 93. At step S245, the noiseremoval section 198 arithmetically operates an average value between thepixel value (luminance candidate value) of the noticed pixel and thesmoothed value a corresponding to the noticed pixel calculated at stepS244 and determines the average value as a pixel value (luminance value)of the luminance image corresponding to the noticed pixel.

The processing returns to step S241 so that the processing at steps S241to S245 is repeated until it is discriminated at step S241 that allpixels have been used as a noticed pixel. When it is discriminated atstep S241 that all pixels have been used as a noticed pixel, the noiseremoval process is ended and also the luminance image production processis ended, and the processing returns to step S212 of FIG. 85.

At step S212, the monochromatic image production sections 182 to 184produce the output images R, G, and B, respectively by using thesupplied color and sensitivity mosaic image and the luminance image.

A first monochromatic image production process of the monochromaticimage production section 182 is described with reference to a flow chartof FIG. 95.

At step S261, the interpolation section 201 performs an interpolationprocess for the color and sensitivity mosaic image to produce an Rcandidate image wherein all pixels have pixel values of an R componentand outputs the R candidate image to the ratio value calculation section202.

It is to be noted that the interpolation process of the interpolationsection 201 is similar to the R component estimation process of theestimation section 191 which composes the luminance image productionsection 181 described hereinabove with reference to the flow chart ofFIG. 87, and therefore, description of it is omitted.

At step S262, the ratio value calculation section 202 performs a ratiovalue calculation process to calculate an intensity ratio and furtherproduces ratio value information representative of the intensity ratiocorresponding to each pixel, and supplies the intensity ratio and theratio value information to the multiplier 203.

The ratio value calculation process of the ratio value calculationsection 202 is described with reference to a flow chart of FIG. 96. Atstep S271, the ratio value calculation section 202 discriminates whetheror not all pixels of the R candidate image have been used as a noticedpixel. If the ratio value calculation section 202 discriminates that allpixels have not been used as a noticed pixel, then the processingadvances to step S272. At step S272, the ratio value calculation section202 determines one by one pixel as a noticed pixel beginning with theleft lowermost pixel and ending with the right uppermost pixel of the Rcandidate image.

At step S273, the ratio value calculation section 202 refers to thosepixels which are positioned in the neighborhood of the noticed pixel(for example, 7×7 pixels centered at the noticed pixel) to acquire thepixel values (monochromatic candidate values of R components) of thepixels. Further, the ratio value calculation section 202 extracts thepixel values (luminance values) of the pixels of the luminance imagewhich are positioned at the same coordinates as those of the referencepixels.

At step S274, the ratio value calculation section 202 acquires a numberof smoothing filter coefficients set in advance as shown in FIG. 97corresponding to relative positions of the reference pixels to thenoticed pixel, the number being equal to the number of the referencepixels.

At step S275, the ratio value calculation section 202 multiplies themonochromatic candidate values for an R component of the referencepixels and the corresponding filter coefficients, divides the productsby the corresponding luminance values and arithmetically operates thesum total of the quotients. Further, the ratio value calculation section202 divides the sum total of the quotients by the sum total of the usedsmoothing filter coefficients and determines the quotient as anintensity ratio corresponding to the noticed pixel to produce ratiovalue information.

The processing returns to step S271 so that the processing at steps S271to S275 is repeated until it is discriminated at step S271 that allpixels of the R candidate image have been used as a noticed pixel. Whenit is discriminated at step S271 that all pixels of the R candidateimage have been used as a noticed pixel, the ratio value informationproduced is supplied to the multiplier 203, and the processing returnsto step S263 of FIG. 95.

At step S263, the multiplier 203 multiplies the pixel values of thepixels of the luminance image by the corresponding intensity ratios toproduce an output image R wherein the products are used as pixel values.

It is to be noted that, simultaneously with the first monochromaticimage production process of the monochromatic image production section182, also the monochromatic image production sections 183 and 184execute similar processes.

Description of the fourth demosaic process by the fourth example of theconfiguration of the image processing section 7 is ended therewith.

FIG. 98 shows a second example of a configuration of the luminance imageproduction section 181. The second example of the configuration of theluminance image production section 181 replaces the estimation sections191 to 193 of the first example of the configuration of the luminanceimage production section 181 shown in FIG. 83 with an estimation section211.

In the second example of the configuration of the luminance imageproduction section 181, a color and sensitivity mosaic image, colormosaic pattern information and sensitivity mosaic pattern informationare supplied to the estimation section 211.

The estimation section 121 performs a component estimation process forthe color and sensitivity mosaic image and supplies an estimation valueR′ of an R component, a estimation value G′ of a G component and anestimation value B′ of a B component for each pixel obtained by thecomponent estimation process to the corresponding multipliers 194 to196, respectively.

It is to be noted that the elements from the multiplier 194 to the noiseremoval section 198 included in the second example of the configurationof the luminance image production section 181 are similar to theelements from the multiplier 194 to the noise removal section 198included in the first example of the configuration of the luminanceimage production section 181 shown in FIG. 83 in which like referencenumerals are applied, and therefore, description of them is omitted.

Now, the estimation process for RGB components by the estimation section211 is described with reference to a flow chart of FIG. 99. It is to benoted that the estimation process for RGB components is a process whichcan be executed in place of the R component estimation process describedhereinabove with reference to FIG. 87 as a process at step S223 of FIG.86. Accordingly, the processing at steps S281 et seq. is describedassuming that a noticed pixel of a color and sensitivity mosaic imagehas already been determined by the estimation section 211.

At step S281, the estimation section 211 calculates an estimated pixelvalue C0 corresponding to the noticed pixel through an estimated pixelvalue C0 interpolation process wherein the pixel values of such fourpixels centered at the noticed pixel as shown in FIG. 100 are used. Theestimated pixel value C0 interpolation process is described withreference to a flow chart of FIG. 101.

At step S291, the estimation section 211 substitutes the pixel values ofthe four pixels positioned upwardly, downwardly, leftwardly andrightwardly of the noticed pixel indicated by ◯ each with a space of onepixel left therebetween into variables a3, a0, a1 and a2 and applies adirection selective smoothing process described hereinabove withreference to FIG. 94 to arithmetically operate a smoothed value α.

The process of substituting the pixel values of four pixels positionedupwardly, downwardly, leftwardly and rightwardly of a designated pixelinto the variables a3, a0, a1 and a2 and applying the directionselective smoothing process described hereinabove with reference to FIG.94 to arithmetically operate a smoothed value α in this manner ishereinafter defined as a vertical direction selective smoothing processcorresponding to the designated pixel.

At step S292, the estimation section 211 adds the smoothed value αobtained at step S291 to the pixel value of the noticed pixel anddetermines the sum as the estimated pixel value C0 of the noticed pixel.The processing returns to step S282 of FIG. 99.

At step S282, the estimation section 211 calculates an estimated pixelvalue C1 corresponding to the noticed pixel through an estimated pixelvalue C1 interpolation process wherein such 12 pixels centered at thenoticed pixel as shown in FIG. 102 are used. The estimated pixel valueC1 interpolation process is described with reference to a flow chart ofFIG. 103.

At step S301, the estimation section 211 discriminates whether or notthe color of the noticed pixel is G. If the estimation section 211discriminates that the color of the noticed pixel is G, then theprocessing advances to step S302. At step S302, the estimation section211 substitutes the pixel values of four pixels positioned leftwardlydownwards, leftwardly upwards, rightwardly downwards and rightwardlyupwards in the neighborhood of the noticed pixel represented by ◯ asshown in FIG. 102 into the variables a0, a1, a2 and a3, respectively,and applies the direction selective smoothing process describedhereinabove with reference to FIG. 94 to arithmetically operate asmoothed value α.

The process of substituting the pixel values of four pixels positionedleftwardly downwards, leftwardly upwards, rightwardly downwards andrightwardly upwards in the neighborhood of a designated pixel into thevariables a0, a1, a2 and a3, respectively, and applying the directionselective smoothing process described hereinabove with reference to FIG.94 to arithmetically operate a smoothed value α is hereinafter definedas an oblique direction selective smoothing process corresponding to thedesignated pixel.

At step S303, the estimation section 211 multiplies the smoothed value αobtained at step S302 by 2 and determines the product as an estimatedpixel value C1 of the noticed pixel. The processing returns to step S283of FIG. 99.

It is to be noted that, if it is discriminated at step S301 that thecolor of the noticed pixel is not G, then the processing advances tostep S304.

At step S304, the estimation section 211 executes the vertical directionselective smoothing process using four pixels positioned with a space ofone pixel left from the pixel neighboring leftwardly upwards of thenoticed pixel to calculate a smoothed value α and substitutes thesmoothed value α into the variable a1. At step S305, the estimationsection 211 executes the vertical direction selective smoothing processusing four pixels positioned with a space of one pixel left from thepixel neighboring rightwardly downwards of the noticed pixel tocalculate a smoothed value α and substitutes the smoothed value α intothe variable a2. At step S306, the estimation section 211 substitutesthe pixel value of the pixel neighboring leftwardly downwards of thenoticed pixel into the variable a0 and substitutes the pixel value ofthe pixel neighboring rightwardly upwards of the noticed pixel into thevariable a3.

At step S307, the estimation section 211 applies the variables a0, a1,a2 and a3 whose values have been set at steps S304 to S306 to thedirection selective smoothing process described hereinabove withreference to FIG. 94 to arithmetically operate a smoothed value α anddetermines the value of the smoothed value α as a smoothed value α′.

At step S308, the estimation section 211 executes the vertical directionselective smoothing process using four pixels positioned with a space ofone pixel left from the pixel neighboring leftwardly downwards of thenoticed pixel to calculate a smoothed value α and substitutes thesmoothed value α into the variable a0. At step S309, the estimationsection 211 executes the vertical direction selective smoothing processusing four pixels positioned with a space of one pixel left from thepixel neighboring rightwardly upwards of the noticed pixel to calculatea smoothed value α and substitutes the smoothed value α into thevariable a3. At step S310, the estimation section 211 substitutes thepixel value of the pixel neighboring leftwardly upwards of the noticedpixel into the variable a1 and substitutes the pixel value of the pixelneighboring rightwardly downwards of the noticed pixel into the variablea2.

At step S311, the estimation section 211 applies the variables a0, a1,a2 and a3 whose values have been set at steps S308 to S310 to thedirection selective smoothing process described hereinabove withreference to FIG. 94 to arithmetically operate a smoothed value α anddetermines the value of the smoothed value α as a smoothed value α″.

At step S312, the estimation section 211 adds the smoothed value α′obtained at step S307 and the smoothed value α″ obtained at step S311and determines the sum as an estimated pixel value C1 corresponding tothe noticed pixel. The processing returns to step S283 of FIG. 99.

At step S283, the estimation section 211 calculates a estimated pixelvalue C2 corresponding to the noticed pixel through a estimated pixelvalue C2 interpolation process wherein such four pixels centered at thenoticed pixel as shown in FIG. 104A or such eight pixels centered at thenoticed pixel as shown in FIG. 104B are used. The estimated pixel valueC2 interpolation process is described with reference to a flow chart ofFIG. 105.

At step S321, the estimation section 211 discriminates whether or notthe color of the noticed pixel is G. If the estimation section 211discriminates that the color of the noticed pixel is G, then theprocessing advances to step S322.

At step S322, the estimation section 211 executes the vertical directionselective smoothing process using four pixels positioned with a space ofone pixel left from the pixel neighboring upwardly of the noticed pixelto calculate a smoothed value α and determines it as a smoothed valueα′.

At step S323, the estimation section 211 executes the vertical directionselective smoothing process using four pixels positioned with a space ofone pixel left from the pixel neighboring downwardly of the noticedpixel to calculate a smoothed value α and determines it as a smoothedvalue α″.

At step S324, the estimation section 211 adds an average value of thepixel value of the pixel neighboring downwardly of the noticed pixel andthe smoothed value α′ obtained at step S322 and an average value of thepixel value of the pixel neighboring upwardly of the noticed pixel andthe smoothed value α″ obtained at step S323 and determines the sum as anestimated pixel value C2 corresponding to the noticed pixel. Theprocessing returns to step S284 of FIG. 99.

It is to be noted that, if it is discriminated at step S321 that thecolor of the noticed pixel is not G, then the processing advances tostep S325.

At step S325, the estimation section 211 executes the oblique directionselective smoothing process using four pixels positioned obliquely inthe neighborhood of the pixel neighboring leftwardly of the noticedpixel to calculate a smoothed value α and substitutes it into thevariable a1. At step S326, the estimation section 211 executes theoblique direction selective smoothing process using four pixelspositioned obliquely in the neighborhood of the pixel neighboringrightwardly of the noticed pixel to calculate a smoothed value α andsubstitutes it into the variable a2. At step S327, the estimationsection 211 substitutes the pixel value of the pixel neighboringdownwardly of the noticed pixel into the variable a0 and substitutes thepixel value of the pixel neighboring upwardly of the noticed pixel intothe variable a3.

At step S328, the estimation section 211 applies the variables a0, a1,a2 and a3 whose values have been set at steps S325 to S327 to thedirection selective smoothing process described hereinabove withreference to FIG. 94 to arithmetically operate a smoothed value α anddetermines the value of the smoothed value α as a smoothed value α′.

At step S329, the estimation section 211 executes the oblique directionselective smoothing process using four pixels positioned obliquely inthe neighborhood of the pixel neighboring downwardly of the noticedpixel to calculate a smoothed value α and substitutes it into thevariable a0. At step S330, the estimation section 211 executes theoblique direction selective smoothing process using four pixelspositioned obliquely in the neighborhood of the pixel neighboringupwardly of the noticed pixel to calculate a smoothed value α andsubstitutes it into the variable a3. At step S331, the estimationsection 211 substitutes the pixel value of the pixel neighboringleftwardly of the noticed pixel into the variable a1 and substitutes thepixel value of the pixel neighboring rightwardly of the noticed pixelinto the variable a2.

At step S332, the estimation section 211 applies the variables a0, a1,a2 and a3 whose values have been set at steps S329 to S331 to thedirection selective smoothing process described hereinabove withreference to FIG. 94 to arithmetically operate a smoothed value α anddetermines the value of the smoothed value α as a smoothed value α″.

At step S333, the estimation section 211 adds the smoothed value α′obtained at step S328 and the smoothed value α″ obtained at step S322and determines the sum as an estimated pixel value C2 corresponding tothe noticed pixel. The processing returns to step S284 of FIG. 99.

At step S284, the estimation section 211 calculates a estimated pixelvalue C3 corresponding to the noticed pixel through an estimated pixelvalue C3 interpolation process wherein such eight pixels centered at thenoticed pixel as shown in FIG. 106 are used. The estimated pixel valueC3 interpolation process is described with reference to a flow chart ofFIG. 107.

At step S341, the estimation section 211 discriminates whether or notthe color of the noticed pixel is G. If the estimation section 211discriminates that the color of the noticed pixel is G, then theprocessing advances to step S342.

At step S342, the estimation section 211 executes the vertical directionselective smoothing process using four pixels positioned with a space ofone pixel left from the pixel neighboring rightwardly of the noticedpixel to calculate a smoothed value α and determines it as a smoothedvalue α′.

At step S343, the estimation section 211 executes the vertical directionselective smoothing process using four pixels positioned with a space ofone pixel left from the pixel neighboring leftwardly of the noticedpixel to calculate a smoothed value α and determines it as a smoothedvalue α″.

At step S344, the estimation section 211 adds an average value of thepixel value of the pixel neighboring leftwardly of the noticed pixel andthe smoothed value α′ obtained at step S342 and an average value of thepixel value of the pixel neighboring rightwardly of the noticed pixeland the smoothed value α″ obtained at step S343 and determines the sumas an estimated pixel value C3 corresponding to the noticed pixel. Theprocessing returns to step S285 of FIG. 99.

It is to be noted that, if it is discriminated at step S341 that thecolor of the noticed pixel is G, then the processing advances to stepS345. At step S345, the estimation section 211 sets the estimated pixelvalue C3 corresponding to the noticed pixel to 0. The processing returnsto step S285 of FIG. 99.

At step S285, the estimation section 211 refers to the color mosaicpattern information and the sensitivity mosaic pattern information todiscriminate the color and the sensitivity of the noticed pixel, andapplies, based on a result of the discrimination, the estimated pixelvalues C0 to C3 corresponding to the noticed pixel obtained at stepsS281 to S284 to a synthetic sensitivity compensation LUT (similar to thesynthetic sensitivity compensation LUT described hereinabove withreference to FIGS. 90 to 92) built therein to calculate estimated valuesR′, G′ and B′.

In particular, where the color of the noticed pixel is G and thesensitivity is S0, a value LUT(C2) when the estimated pixel value C2 isapplied to the synthetic sensitivity compensation LUT is determined asthe estimated value R′, and a value LUT((C0+C1/)2)) when an averagevalue of the estimated pixel values C0+C1 is applied to the syntheticsensitivity compensation LUT is determined as the estimated value G′while a value LUT(C3) when the estimated pixel value C3 is applied tothe synthetic sensitivity compensation LUT is determined as theestimated value B′.

Where the color of the noticed pixel is G and the sensitivity is S1, avalue LUT(C3) when the estimated pixel value C3 is applied to thesynthetic sensitivity compensation LUT is determined as the estimatedvalue R′, and a value LUT((C0+C1/)2)) when an average value of theestimated pixel values C0+C1 is applied to the synthetic sensitivitycompensation LUT is determined as the estimated value G′ while a valueLUT(C2) when the estimated pixel value C2 is applied to the syntheticsensitivity compensation LUT is determined as the estimated value B′.

Where the color of the noticed pixel is R, a value LUT(C0) when theestimated pixel value C0 is applied to the synthetic sensitivitycompensation LUT is determined as the estimated value R′, and a valueLUT(C2) when an average value of the estimated pixel value C2 is appliedto the synthetic sensitivity compensation LUT is determined as theestimated value G′ while a value LUT(C1) when the estimated pixel valueC1 is applied to the synthetic sensitivity compensation LUT isdetermined as the estimated value B′.

Where the color of the noticed pixel is B, a value LUT(C1) when theestimated pixel value C1 is applied to the synthetic sensitivitycompensation LUT is determined as the estimated value R′, and a valueLUT(C2) when an average value of the estimated pixel value C2 is appliedto the synthetic sensitivity compensation LUT is determined as theestimated value G′ while a value LUT(C0) when the estimated pixel valueC0 is applied to the synthetic sensitivity compensation LUT isdetermined as the estimated value B′.

Since, in the estimation process of RGB components by the estimationsection 211, the estimated pixel values C0 to C3 produced making use ofthe direction selective smoothing process are used in such a manner asdescribed above, deterioration of the resolution of an image signal issuppressed.

Description of the estimation process for RGB components by theestimation section 211 is ended therewith.

Incidentally, it is described in the foregoing description that themonochromatic image production sections 183 and 184 of the fourthexample of the configuration of the image processing section 7 areconfigured similarly to the example of the configuration of themonochromatic image production section 182 shown in FIG. 84 and executea process similar to the monochromatic image production process (FIG.95) of the monochromatic image production section 182 described withreference to FIG. 95. However, the monochromatic image productionsections 182 to 184 may otherwise execute unique processes individuallyoptimized therefor in place of the monochromatic candidate image process(step S261 of FIG. 95) included in the monochromatic image productionprocess.

The R candidate image production process executed by the monochromaticimage production section 182 in place of the monochromatic candidateimage production process at step S261 is described with reference to aflow chart of FIG. 108. It is to be noted that, for the convenience ofdescription, the interpolation section 201 which composes themonochromatic image production section 182 is hereinafter referred to asinterpolation section 201-R.

At step S351, the interpolation section 201-R discriminates whether ornot all pixels of the color and sensitivity mosaic image have been usedas a noticed pixel for the first time. If the interpolation section201-R discriminates that all pixels have not been used as a noticedpixel for the first time, then the processing advances to step S352. Atstep S352, the interpolation section 201-R determines one by one pixelas a noticed pixel for the first time beginning with the left lowermostpixel and ending with the right uppermost pixel of the color andsensitivity mosaic image.

At step S353, the interpolation section 201-R discriminates whether ornot the color of the noticed pixel for the first time is R. If theinterpolation section 201-R discriminates that the color of the noticedpixel for the first time is R, then the processing advances to stepS354. At step S354, the interpolation section 201-R executes thevertical direction selective smoothing process using four pixelspositioned upwardly, downwardly, leftwardly and rightwardly of thenoticed pixel for the first time with a space of one pixel lefttherebetween to calculate a smoothed value α. At step S355, theinterpolation section 201-R applies the sum of the pixel value of thenoticed pixel for the first time and the smoothed value α calculated atstep S354 to a synthetic sensitivity compensation LUT (a syntheticsensitivity compensation LUT similar to that described with reference toFIGS. 90 to 92) built therein and determines the resulting value as apixel value corresponding to the noticed pixel for the first time of anR candidate image. The processing returns to step S351.

It is to be noted that, if it is discriminated at step S353 that thecolor of the noticed pixel for the first time is not R, then theprocessing returns to step S351 skipping the steps S354 and S355.

Thereafter, the processing at steps S351 to S355 is repeated until it isdiscriminated at step S351 that all pixels of the color and sensitivitymosaic image have been used as a noticed pixel for the first time. Whenit is discriminated at step S351 that all pixels of the color andsensitivity mosaic image have been used as a noticed pixel for the firsttime, the processing advances to step S356.

At step S356, the interpolation section 201-R discriminates whether ornot all pixels of the color and sensitivity mosaic image have been usedas a noticed pixel for the second time. If the interpolation section201-R discriminates that all pixels have not been used as a noticedpixel for the second time, then the processing advances to step S357. Atstep S357, the interpolation section 201-R determines one by one pixelas a noticed pixel for the second time beginning with the left lowermostpixel and ending with the right uppermost pixel of the color andsensitivity mosaic image.

At step S358, the interpolation section 201-R discriminates whether ornot the color of the noticed pixel for the second time is B. If theinterpolation section 201-R discriminates that the color of the noticedpixel for the second time is B, then the processing advances to stepS359. At step S359, the interpolation section 201-R executes the obliquedirection selective smoothing process using four pixels positionedobliquely in the neighborhood of the noticed pixel for the second timeto calculate a smoothed value α. At step S360, the interpolation section201-R determines the smoothed value α calculated at step S359 as a pixelvalue corresponding to the noticed pixel for the second time of the Rcandidate image. The processing returns to step S356.

It is to be noted that, if it is discriminated at step S358 that thecolor of the noticed pixel for the second time is not B, then theprocessing returns to step S356 skipping the steps S359 and S360.

Thereafter, the processing at steps S356 to S360 is repeated until it isdiscriminated at step S356 that all pixels of the color and sensitivitymosaic image have been used as a noticed pixel for the second time. Whenit is discriminated at step S356 that all pixels of the color andsensitivity mosaic image have been used as a noticed pixel for thesecond time, the processing advances to step S351.

At step S361, the interpolation section 201-R discriminates whether ornot all pixels of the color and sensitivity mosaic image have been usedas a noticed pixel for the third time. If the interpolation section201-R discriminates that all pixels have not been used as a noticedpixel for the third time, then the processing advances to step S362. Atstep S362, the interpolation section 201-R determines one by one pixelas a noticed pixel for the third time beginning with the left lowermostpixel and ending with the right uppermost pixel of the color andsensitivity mosaic image.

At step S363, the interpolation section 201-R discriminates whether ornot the color of the noticed pixel for the third time is G. If theinterpolation section 201-R discriminates that the color of the noticedpixel for the third time is G, then the processing advances to stepS364. At step S364, the interpolation section 201-R executes thevertical direction selective smoothing process using four pixelspositioned upwardly, downwardly, leftwardly and rightwardly of thenoticed pixel for the third time to calculate a smoothed value α. Atstep S365, the interpolation section 201-R determines the smoothed valueα calculated at step S364 as a pixel value corresponding to the noticedpixel for the third time of an R candidate image.

It is to be noted that, if it is discriminated at step S363 that thecolor of the noticed pixel for the third time is not G, then theprocessing returns to step S351 skipping the steps S364 and S365.

Thereafter, the processing at steps S361 to S365 is repeated until it isdiscriminated at step S361 that all pixels of the color and sensitivitymosaic image have been used as a noticed pixel for the third time. Whenit is discriminated at step S361 that all pixels of the color andsensitivity mosaic image have been used as a noticed pixel for the thirdtime, the R candidate image production process is ended.

The B candidate image production process executed by the monochromaticimage production section 184 is described with reference to a flow chartof FIG. 109. It is to be noted that, for the convenience of description,the component of the monochromatic image production section 184 whichcorresponds to the interpolation section 201 of the monochromatic imageproduction section 182 is hereinafter referred to as interpolationsection 201-B.

At step S371, the interpolation section 201-B discriminates whether ornot all pixels of the color and sensitivity mosaic image have been usedas a noticed pixel for the first time. If the interpolation section201-B discriminates that all pixels have not been used as a noticedpixel for the first time, then the processing advances to step S372. Atstep S372, the interpolation section 201-B determines one by one pixelas a noticed pixel for the first time beginning with the left lowermostpixel and ending with the right uppermost pixel of the color andsensitivity mosaic image.

At step S373, the interpolation section 201-B discriminates whether ornot the color of the noticed pixel for the first time is B. If theinterpolation section 201-B discriminates that the color of the noticedpixel for the first time is B, then the processing advances to stepS374. At step S374, the interpolation section 201-B executes thevertical direction selective smoothing process using four pixelspositioned upwardly, downwardly, leftwardly and rightwardly of thenoticed pixel for the first time with a space of one pixel lefttherebetween to calculate a smoothed value α. At step S375, theinterpolation section 201-B applies the sum of the pixel value of thenoticed pixel for the first time and the smoothed value α calculated atstep S374 to a synthetic sensitivity compensation LUT (a syntheticsensitivity compensation LUT similar to that described with reference toFIGS. 90 to 92) built therein and determines the resulting value as apixel value corresponding to the noticed pixel for the first time of a Bcandidate image. The processing returns to step S371.

It is to be noted that, if it is discriminated at step S373 that thecolor of the noticed pixel for the first time is not B, then theprocessing returns to step S371 skipping the steps S374 and S375.

Thereafter, the processing at steps S371 to S375 is repeated until it isdiscriminated at step S371 that all pixels of the color and sensitivitymosaic image have been used as a noticed pixel for the first time. Whenit is discriminated at step S371 that all pixels of the color andsensitivity mosaic image have been used as a noticed pixel for the firsttime, the processing advances to step S376.

At step S376, the interpolation section 201-B discriminates whether ornot all pixels of the color and sensitivity mosaic image have been usedas a noticed pixel for the second time. If the interpolation section201-B discriminates that all pixels have not been used as a noticedpixel for the second time, then the processing advances to step S377. Atstep S377, the interpolation section 201-B determines one by one pixelas a noticed pixel for the second time beginning with the left lowermostpixel and ending with the right uppermost pixel of the color andsensitivity mosaic image.

At step S378, the interpolation section 201-B discriminates whether ornot the color of the noticed pixel for the second time is R. If theinterpolation section 201-B discriminates that the color of the noticedpixel for the second time is R, then the processing advances to stepS379. At step S379, the interpolation section 201-B executes the obliquedirection selective smoothing process using four pixels positionedobliquely in the neighborhood of the noticed pixel for the second timeto calculate a smoothed value α. At step S380, the interpolation section201-B determines the smoothed value α calculated at step S379 as a pixelvalue corresponding to the noticed pixel for the second time of the Bcandidate image. The processing returns to step S376.

It is to be noted that, if it is discriminated at step S378 that thecolor of the noticed pixel for the second time is not R, then theprocessing returns to step S376 skipping the steps S379 and S380.

Thereafter, the processing at steps S376 to S380 is repeated until it isdiscriminated at step S376 that all pixels of the color and sensitivitymosaic image have been used as a noticed pixel for the second time. Whenit is discriminated at step S376 that all pixels of the color andsensitivity mosaic image have been used as a noticed pixel for thesecond time, the processing advances to step S381.

At step S381, the interpolation section 201-B discriminates whether ornot all pixels of the color and sensitivity mosaic image have been usedas a noticed pixel for the third time. If the interpolation section201-B discriminates that all pixels have not been used as a noticedpixel for the third time, then the processing advances to step S382. Atstep S382, the interpolation section 201-B determines one by one pixelas a noticed pixel for the third time beginning with the left lowermostpixel and ending with the right uppermost pixel of the color andsensitivity mosaic image.

At step S383, the interpolation section 201-B discriminates whether ornot the color of the noticed pixel for the third time is G. If theinterpolation section 201-B discriminates that the color of the noticedpixel for the third time is G, then the processing advances to stepS384. At step S384, the interpolation section 201-B executes thevertical direction selective smoothing process using four pixelspositioned upwardly, downwardly, leftwardly and rightwardly in theneighborhood of the noticed pixel for the third time to calculate asmoothed value α. At step S385, the interpolation section 201-Bdetermines the smoothed value α calculated at step S384 as a pixel valuecorresponding to the noticed pixel for the third time of a B candidateimage. The processing returns to step S381.

It is to be noted that, if it is discriminated at step S383 that thecolor of the noticed pixel for the third time is not G, then theprocessing returns to step S381 skipping the steps S384 and S385.

Thereafter, the processing at steps S381 to S385 is repeated until it isdiscriminated at step S381 that all pixels of the color and sensitivitymosaic image have been used as a noticed pixel for the third time. Whenit is discriminated at step S381 that all pixels of the color andsensitivity mosaic image have been used as a noticed pixel for the thirdtime, the B candidate image production process is ended.

The G candidate image production process executed by the monochromaticimage production section 183 is described with reference to a flow chartof FIG. 110. It is to be noted that, for the convenience of description,the component of the monochromatic image production section 183 whichcorresponds to the interpolation section 201 of the monochromatic imageproduction section 182 is hereinafter referred to as interpolationsection 201-G.

At step S391, the interpolation section 201-G discriminates whether ornot all pixels of the color and sensitivity mosaic image have been usedas a noticed pixel for the first time. If the interpolation section201-G discriminates that all pixels have not been used as a noticedpixel for the first time, then the processing advances to step S392. Atstep S392, the interpolation section 201-G determines one by one pixelas a noticed pixel for the first time beginning with the left lowermostpixel and ending with the right uppermost pixel of the color andsensitivity mosaic image.

At step S393, the interpolation section 201-G discriminates whether ornot the color of the noticed pixel for the first time is G. If theinterpolation section 201-G discriminates that the color of the noticedpixel for the first time is G, then the processing advances to stepS394. At step S394, the interpolation section 201-G executes the obliquedirection selective smoothing process using four pixels positionedobliquely in the neighborhood of the noticed pixel for the first time tocalculate a smoothed value α. At step S395, the interpolation section201-G applies the sum of the pixel value of the noticed pixel for thefirst time and the smoothed value α calculated at step S394 to asynthetic sensitivity compensation LUT (a synthetic sensitivitycompensation LUT similar to that described with reference to FIGS. 90 to92) built therein and determines the resulting value as a pixel valuecorresponding to the noticed pixel for the first time of a G candidateimage. The processing returns to step S391.

It is to be noted that, if it is discriminated at step S393 that thecolor of the noticed pixel for the first time is not G, then theprocessing returns to step S391 skipping the steps S394 and S395.

Thereafter, the processing at steps S391 to S395 is repeated until it isdiscriminated at step S391 that all pixels of the color and sensitivitymosaic image have been used as a noticed pixel for the first time. Whenit is discriminated at step S391 that all pixels of the color andsensitivity mosaic image have been used as a noticed pixel for the firsttime, the processing advances to step S396.

At step S396, the interpolation section 201-G discriminates whether ornot all pixels of the color and sensitivity mosaic image have been usedas a noticed pixel for the second time. If the interpolation section201-G discriminates that all pixels have not been used as a noticedpixel for the second time, then the processing advances to step S397. Atstep S397, the interpolation section 201-G determines one by one pixelas a noticed pixel for the second time beginning with the left lowermostpixel and ending with the right uppermost pixel of the color andsensitivity mosaic image.

At step S398, the interpolation section 201-G discriminates whether ornot the color of the noticed pixel for the second time is G. If theinterpolation section 201-G discriminates that the color of the noticedpixel for the second time is not G, then the processing advances to stepS399. At step S399, the interpolation section 201-G executes thevertical direction selective smoothing process using four pixelspositioned upwardly, downwardly, leftwardly and rightwardly in theneighborhood of the noticed pixel for the second time to calculate asmoothed value α. At step S400, the interpolation section 201-Gdetermines the smoothed value α calculated at step S399 as a pixel valuecorresponding to the noticed pixel for the second time of the Gcandidate image. The processing returns to step S396.

It is to be noted that, if it is discriminated at step S398 that thecolor of the noticed pixel for the second time is R, then the processingreturns to step S396 skipping the steps S399 and S400.

Thereafter, the processing at steps S396 to S400 is repeated until it isdiscriminated at step S396 that all pixels of the color and sensitivitymosaic image have been used as a noticed pixel for the second time. Whenit is discriminated at step S396 that all pixels of the color andsensitivity mosaic image have been used as a noticed pixel for thesecond time, the R candidate image production process is ended.

Incidentally, as described hereinabove, in the fourth demosaic process,a luminance image and monochromatic images are produced from a color andsensitivity mosaic image, and all colors are restored making use of thecorrelation between the luminance and the color components to restoreall pixels having a uniform sensitivity and all color components.However, the luminance image to be produced first may have a biasedspectral characteristic only if color information to be restored has thecorrelation and the signal can be restored with a high resolution. Forexample, the characteristic of a color mosaic arrangement of a color andsensitivity mosaic image that it includes a number of pixels of G equalto twice that of pixels of R or pixels of B like a Bayer arrangement maybe utilized to produce an image of a G component in place of a luminanceimage, and the correlation between G and R or between G and B may beutilized to produce an image of an R component and an image of a Bcomponent.

To execute such processing as just described, the image processingsection 7 may be configured in such a manner as shown in FIG. 110. Aluminance image production section 221 executes processing similar tothat of the interpolation section 201 (FIG. 84) of the monochromaticimage production section 182 in the fourth example of the configurationof the image processing section 7 to produce an output image G.Monochromatic image production sections 222 and 223 execute processingsimilar to that of the monochromatic image production sections 182 and184 in the fourth example of the configuration of the image processingsection 7 to produce an output image R and an output image B,respectively.

Description of the examples of the configuration of the image processingsection 7 for executing the first to fourth demosaic processes is endedtherewith.

It is to be noted that, while the series of processes described abovecan be executed by hardware, it may otherwise be executed by software.Where the series of processes is executed by software, a program whichconstructs the software is installed from a recording medium into acomputer incorporated in hardware for exclusive use or, for example, apersonal computer for universal use which can execute various functionsby installing various programs.

The recording medium is formed as a package medium such as, as shown inFIG. 1, a magnetic disc 16 (including a floppy disc), an optical disc 17(including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (DigitalVersatile Disc)), or a magneto-optical disc 18 (including an MD (MiniDisc)), or a semiconductor memory 19 which has the program recordedthereon or therein and is distributed to provide the program to a userseparately from a computer. Else, the recording medium is formed as aROM, a hard disc or the like in which the program is recorded and whichis provided to a user in a state wherein the program is incorporated ina computer.

It is to be noted that, in the present specification, the steps whichdescribe the program recorded in a recording medium may be but need notnecessarily be processed in a time series in the order as described, andinclude processes which are executed in parallel or individually withoutbeing processed in a time series.

INDUSTRIAL APPLICABILITY

As described above, according to the present invention, a restored imagewherein the sensitivities of pixels are uniformized and each pixel hasall of a plurality of color components.

It should be understood that various changes and modifications to thepresently preferred embodiments described herein will be apparent tothose skilled in the art. Such changes and modifications can be madewithout departing from the spirit and scope of the present subjectmatter and without diminishing its intended advantages. It is thereforeintended that such changes and modifications be covered by the appendedclaims.

The invention claimed is as follows:
 1. An image processing apparatuscomprising: restoration means for generating a restoration image basedon a color-and-sensitivity mosaic image wherein (a) each of a pluralityof pixels has one of first to third color components and one of aplurality of sensitivity characteristics with respect to intensity oflight, (b) in terms of the sensitivity characteristics of the pixels,the pixels disposed in a same line have a same sensitivitycharacteristic, and the pixels disposed in different lines, which areadjacent to each other, have different sensitivity characteristics, and(c) in terms of the color components of the pixels, the pixels havingthe first color component are arranged in a checker pattern irrespectiveof the pixels' sensitivity characteristics, the pixels having the secondcolor component are arranged so as to be disposed in adjacent differentlines and adjacent in a diagonal direction, and the pixels having thethird color component are arranged so as to be disposed in adjacentdifferent lines and adjacent in a diagonal direction, the restorationmeans including: in the restoration image, for each color component, thesensitivities of the pixels are uniformized, using as inputs (i) colormosaic pattern information, (ii) a color and sensitivity mosaic image,and (iii) sensitivity mosaic pattern information, so thatsensitivitycompensation means using a color and sensitivity mosaic image as aninput for scaling each of the pixels having different sensitivitycharacteristics are (i) scaled to theto a same light intensity as thepixels having thea same sensitivity characteristicscharacteristic tocreate sensitivity compensated pixel information for each of the pixelshaving different sensitivity characteristics; and (ii) comparedvaliditydiscrimination means using the color and sensitivity mosaic image as aninput for comparing each of the pixels having different sensitivitycharacteristics with a threshold value to discriminate the validity ofthe pixel value to create discrimination information for each of thepixels having different sensitivity characteristics, and each of theuniformized pixels has all of the plurality of color components; andinterpolation means for interpolating the sensitivity compensated pixelinformation based on the discrimination information to produce therestoration image.
 2. An image processing method comprising: arestoration step of generating a restoration image based ona step forproviding a color-and-sensitivity mosaic image wherein, (a) each of aplurality of pixels has one of first to third color components and oneof a plurality of sensitivity characteristics with respect to intensityof light, (b) in terms of sensitivity characteristics of the pixels, thepixels disposed in a same line have a same sensitivity characteristic,and the pixels disposed in different lines, which are adjacent to eachother, have different sensitivity characteristics, and (c) in terms ofcolor components of the pixels, the pixels having the first colorcomponent are arranged in a checker pattern irrespective of the pixels'sensitivity characteristics, the pixels having the second colorcomponent are arranged so as to be disposed in adjacent different linesand adjacent in a diagonal direction, and the pixels having the thirdcolor component are arranged so as to be disposed in adjacent differentlines and adjacent in a diagonal direction,; in the restoration image,for each color component, the sensitivities of the pixels areuniformized, using as inputs (i) color mosaic pattern information, (ii)a color and sensitivity mosaic image, and (iii) sensitivity mosaicpattern information, so thatusing a color and sensitivity mosaic imageas an input, a step for scaling each of the pixels having differentsensitivity characteristics are (i) scaled to the a same light intensityas the pixels having the a same sensitivity characteristicscharacteristic to create sensitivity compensated pixel information foreach of the pixels having different sensitivity characteristics; and(ii) comparedusing the color and sensitivity mosaic image as an input, astep for comparing each of the pixels having different sensitivitycharacteristics with a threshold value to discriminate the validity ofthe pixel value to create discrimination information for each of thepixels having different sensitivity characteristics, and each of theuniformized pixels has all of the plurality of color components; and astep for interpolating the sensitivity compensated pixel informationbased on the discrimination information to produce a restoration image.3. An image processing apparatus comprising: restoration means forgenerating a restoration image based on a color-and-sensitivity mosaicimage wherein (a) each of a plurality of pixels has one of first tothird color components and one of a plurality of sensitivitycharacteristics with respect to intensity of light, (b) the pixelshaving the first to third color components are arranged in a Bayerpattern with their respective color components, and (c) the pixelshaving the first color component have different sensitivitycharacteristics from each other in different lines, the pixels havingthe second color component are arranged so as to form a checker patternwith their sensitivity characteristics have different sensitivitycharacteristics from each other in different lines, and the pixelshaving the third color component are arranged so as to form a checkerpattern with their respective sensitivity characteristics, and therestoration means including: in the restoration image, for each colorcomponent, the sensitivities of the pixels are uniformized, using asinputs (i) color mosaic pattern information, (ii) a color andsensitivity mosaic image, and (iii) sensitivity mosaic patterninformation, so that sensitivity compensation means using a color andsensitivity mosaic image as an input for scaling each of the pixelshaving different sensitivity characteristics are (i) scaled to the asame light intensity as the pixels having the a same sensitivitycharacteristics characteristic to create sensitivity compensated pixelinformation for each of the pixels having different sensitivitycharacteristics; and (ii) comparedvalidity discrimination means usingthe color and sensitivity mosaic image as an input for comparing each ofthe pixels having different sensitivity characteristics with a thresholdvalue to discriminate the validity of the pixel value to creatediscrimination information for each of the pixels having differentsensitivity characteristics, and each of the uniformized pixels has allof the plurality of color components; and interpolation means forinterpolating the sensitivity compensated pixel information based on thediscrimination information to produce the restoration image.
 4. An imageprocessing method comprising: a restoration step of generating arestoration image based ona step for providing a color-and-sensitivitymosaic image wherein (a) each of a plurality of pixels has one of firstto third color components and one of a plurality of sensitivitycharacteristics with respect to intensity of light, (b) the pixelshaving the first to third color components are arranged in a Bayerpattern with their color components, and (c) the pixels having the firstcolor component have different sensitivity characteristics from eachother in different lines, the pixels having the second color componentare arranged so as to form a checker pattern with their sensitivitycharacteristics have different sensitivity characteristics from eachother in different lines, and the pixels having, the third colorcomponent are arranged so as to form a checker pattern with theirrespective sensitivity characteristics, and; in the restoration image,for each color component, the sensitivities of the pixels areuniformized, using as inputs (i) color mosaic pattern information, (ii)a color and sensitivity mosaic image, and (iii) sensitivity mosaicpattern information, so thatusing a color and sensitivity mosaic imageas an input, a step for scaling each of the pixels having differentsensitivity characteristics are (i) scaled to the a same light intensityas the pixels having the a same sensitivity characteristicscharacteristic to create sensitivity compensated pixel information foreach of the pixels having different sensitivity characteristics; and(ii) comparedusing the color and sensitivity mosaic image as an input, astep for comparing each of the pixels having different sensitivitycharacteristics with a threshold value to discriminate the validity ofthe pixel value to create discrimination information for each of thepixels having different sensitivity characteristics, and each of theuniformized pixels has all of the plurality of color components; and astep for interpolating the sensitivity compensated pixel informationbased on the discrimination information to produce a restoration image.